istqb full guide

8/3/2019 Istqb Full Guide

1/75

Chapter 1

Principles of Testing

1.1 OVERVIEW

In this module you will get an overview of the most fundamental principles of testing.

Firstly, we point out that although you will come across different terminology throughoutyour career, professional testers are all pretty much agreed on these basic ideas that wedescribe in this module.

Secondly, we take a look at the need for proper testing look at the cost of getting it wrongand we show why exhaustive testing is neither possible not practical. What are errors andhow do they get into the software?

Thirdly, we describe a fundamental test process, base on industry standards, and underlineimportance of planning) tests and determining expected results in advance of test execution.

We conclude with a look at re-testing and regression testing; why it is so important to go thatextra mile and run a suite of tests again, just when you thought it was safe to hit the releasedate.

1.2 OBJECTIVES

After completing this module you will:

Understand basic testing terminology. Understand why testing is necessary.

Be able to define error, fault and failure. Appreciate why errors occur and how costly they can be. Understand that you cannot test everything and that testing is therefore a risk managementprocess. Understand the fundamental test process. Understand that developers and testers have different mindsets. Learn how to communicate effectively with both developers and testers. Find out why you cannot test your own work. Understand the need for regression testing. Understand the importance of specifying your expected results in advance. Understand how and why tests should be prioritized.

1


2/75

1.3 TESTING TERMINOLOGY

There is no generally accepted set of testing definitions used by the worldwide testingcommunity. The British Standard for Software Component Testing published in August 1998 provides a new source of testing definitions. When involved in a testing project it isimportant to ensure everyone understands terminology adopted; you may find differentterminology is used in your organization.

Exercise

The British Standard for Software Component Testing is known as BS___________A useful glossary of terms used in software testing is called BS___________

Although it is a British Standard published by the BSI (British Standards Institute), theSpecialist Interest Group in Software Testing (SIGIST) developed it over a period of severalyears.

ANSWERS: 7925-PART2, 7925-PART1

1.4 WHYIS TESTINGNECESSARY?

This section explains why testing is necessary and closely at the cost and consequences oferrors in computer software.

1.4.1 Definitions Errors, Faults and Failures

An error is a human action that produces an incorrect result. A fault is a manifestation of anerror in software. Faults are also known colloquially as defaults or bugs. A fault, ifencountered, may cause a fault, which is a deviation of the software from its existing delivery

or service.

We can illustrate these points with the true story Mercury spacecraft. The computer programaboa spacecraft contained the following statement wri1 the FORTRAN programminglanguage.

DO 100 i = 1.10

The programmer's intention was to execute a succeeding statements up to line 100 ten timesthen creating a loop where the integer variable I was using the loop counter, starting 1 andending at 10.

Unfortunately, what this code actually does is writing variable i do to decimal value 1.1 andit does that once only. Therefore remaining code is executed once and not 10 times within theloop. As a result spacecraft went off course and mission was abort considerable cost!

The correct syntax for what the programmer intended isDO 100 i =1,10

2


3/75

Exercise

What do you think was the error, fault and failure in this example?

The error is __________

The fault is ___________

The failure is __________

1.4.2 Reliability

Reliability is the probability that software will not cause the failure of a system for aspecified time under specified conditions. Measures of reliability include MTBF (mean timebetween failure), MTTF (mean time to failure) as well as service level agreements and othermechanisms.

1.4.3 Errors and how they occur

Why do we make errors that causefaults in computer software leading to potentialfailure ofour systems? Well, firstly we are all prone to making simple human errors. This is anunavoidable fact of life. However, this is compounded by the fact that we all operate underreal world pressures such as tight deadlines, budget restrictions, conflicting priorities and soon.

1.4.4 Cost of errors

The cost of an error can vary from nothing at all to large amounts of money and even loss oflife. The aborted Mercury mission was obviously very costly but surely this is just an isolated

example. Or is it? There are hundreds of stories about failures of computer systems that havebeen attributed to errors in the software. A few examples are shown below:

A nuclearreactor was shut downbecause a single line of code was coded as X = Y insteadof X=ABS (Y) i.e. the absolute value of Y irrespective of whether Y was positive ornegative.

Blue Cross of Wisconsin installed a new $200m claims processing system - it sent out $60million in unwarranted and duplicate payments. Also, when a data entry clerk typed 'none'in the town field the system sent hundreds of checks to the non-existent town of 'NONE' inWisconsin.

In May 1992, Pepsi fan a promotion in the Philippines. It told customers they could win amillion pesos (approx. $40,000) if they bought a bottle of Pepsi and found number 349stamped on the underside of the bottle cap. Unfortunately, due to a software error, 800,000bottle caps were produced with number 349 instead of one, which was an equivalent of $42billion in prize money. It cost the company dearly as some people pursued their claimsthrough the courts and Pepsi paid out millions of dollars in compensation.

Another story was printed in the New York Times on 18th February 1994. Chemical Bank

3


4/75

managed to allow $15 million to be withdrawn incorrectly from 100,000 accounts - a singleline error in the program caused every ATM on their network to process the transactiontwice.

1.4.5 what happened on October 26th & 27th 1992?

The London Ambulance Service (LAS) covers an area of just over 600 square miles and isthe largest ambulance service in the world. It covers a resident population of some 6.8

million, but its daytime population is larger, especially in central London. Since 1974 SouthWest Thames Regional Health Authority has managed it.

The LAS carries over 5000 patients every day and receives between 2000 and 2500 callsdaily (including between 1300 and 1600 emergency calls i.e. 999 calls). In terms of resourcesthe LAS has some 2700 staff, 750 ambulances and a small number of various other vehiclesincluding 1 helicopter. LAS make almost 2 million patient journeys each year. In 1992/1993its budgeted income was 69.7 million.

On the 26th and 27thOctober 1992 the new system failed, ambulances failed to turn up andpeople lost their lives. Although no Coroners Court ever apportioned blame for any deaths

directly to the computer systems failure, it was by any standards a major disaster and madethe main evening news bulletins on several occasions.

1.4.6 London Ambulance Service

In summary the problems were:Computer Aided Dispatch - 1

The system relied on near perfect information to propose optimum resource to be allocated toan emergency. However, there were imperfections in the information and changes inoperational procedures made it difficult for staff to correct the system.

This was not a problem when it went live at 7 am on 26th October 1992 as the system loadwas light; however as the number of emergency calls increased throughout the-day it becameincreasingly difficult for staff to correct errors; this led to:

Poor, duplicated and delayed allocations.

Build-up of exception messages and awaiting attention list.

Slow-down of system as messages and lists built up.

Increased number of callbacks and hence delays in telephone answering.

The cost of these errors were ultimately that ambulances didn't turn up and people lost their

lives although the official enquiry report did not attribute any fatalities to the system problems. The costs in terms of loss of confidence in the computer system, industrialrelations and so on were probably also high.

4


5/75

1.4.7 Exhaustive testing why not test everything?

It is now widely accepted that you cannot test everything. Exhausted testers you will find, butexhaustive testing you will not. Complete testing is neither theoretically, nor practically possible. Consider a 10-character string that has 280 possible input streams andcorresponding outputs. If you executed one test per microsecond it would take approx. 4times the age of the Universe to test this completely. For a survey of the methodology andlimitations of formal proofs of program correctness see [Manna 78]

1.4.8 Testing and risk

How much testing would you be willing to perform if the risk of failure were negligible?Alternatively, how much testing would you be willing to perform if a single defect could costyou your life's savings, or, even more significantly, your life? [Hetzel 88].

The amount of testing performed depends on the risks involved. Risk must be used as thebasis for allocating the test time that is available and for selecting what to test and where toplace emphasis. A priority must be assigned to each test.

Test Managers and Project Managers come up with different prioritization schemes but htbasic principle is that you must focus the testing effort on those areas of the system that arelikely to have the most defects. Another key principle is that you must execute the mostimportant test first. Otherwise, if you run out of time, which is likely, you will not haveexercised the tests that give you the best payback in terms of faults found.

1.4.9 Testing and quality

Testing identifies faults whose removal increases the software quality by increasing thesoftware's potential reliability. Testing is the measurement of software quality. We measurehow closely we have achieved quality by testing the relevant factors such as correctness,

reliability, usability, maintainability, reusability, testability etc.

1.4.10 Testing and legal, contractual, regulatory or mandatory requirements

Other factors that may determine the testing performed may be legal, contractualrequirements, normally defined in industry specific standards or based on agreed bestpractice (or more realistically non-negligent practice).

1.4.11 How much testing is enough?

It is difficult to determine how much testing is enough. Testing is always a matter of judgingrisks against cost of extra testing effort. Planning test effort thoroughly before you begin, andsetting completion criteria will go some way towards ensuring the right amount of testing isattempted. Assigning priorities to tests will ensure that the most important tests have beendone should you run out of time.

5


6/75

1.5 FUNDAMENTAL TEST PROCESS

1.5.1 Introduction

Testing must be planned. This is one of Bill Hetzel's 6 testing principles [Hetzel 88 p25] andhe says we are all agreed on this one. However, he points out that the problem is that most ofus do not discipline ourselves to act upon it. Good testing requires thinking out an overallapproach, designing tests and establishing expected results' for each of the test cases we

choose.

You have seen already that we cannot test everything, we must make a selection, and theplanning and care we expend on that selection accounts for much of the difference betweengood and poor testers.

The quality and effectiveness of software testing are primarily determined by the quality ofthe test processes used [Kit 95]. This is one of Ed Kit's 6 essentials of software testing. Testgroups that operate within organizations that have an immature development process will feelmore pain than those that do not. However, the test group should strive to improve its owninternal testing processes. This section of the course shows a fundamental test process, based

on the BS7925-2 standard for software component testing.

The fundamental test process comprises planning, specification, execution, recording andchecking for completion. You will find organizations that have slightly different names foreach stage of the process and you may find some processes that have just 4 stages, where 4 &5 are combined, for example. However, you will find that all good test processes adhere tothis fundamental structure.

1.5.2 Test process stages

See BS7925-2 for diagram of test process. Test planning involves producing a document that

describes an overall approach and test objectives noting any assumptions you have made andstating any exceptions to your overall test strategy for your project. Test planning can beapplied at all levels. Completion or exit criteria must be specified so that you know whentesting (at any stage) is complete. Often a coverage target is set and used as test completioncriteria.

Test specification (sometimes referred to as test design) involves designing test conditionsand test cases using recognized test techniques identified at the planning stage. Here it isusual to produce a separate document or documents that fully describe the tests that you willcarry out. It is important to determine the expected resultsprior to test execution.

Test execution involves actually running the specified test on a computer system eithermanually or by using an automated test tool.

Test recording involves keeping good records of the test activities that you have carried out.Versions of the software you have tested and the test specifications are software you havetested and the test specifications are recorded along with the actual outcomes of each test.

Checking for test completion involves looking at the previously specified test completioncriteria to see if they have been met. If not, some test may need to be rerun and in some

6


7/75

instances it may be appropriate to design some new test cases to meet a particular coveragetarget.

Note that BS7925-2 does not specify the format of any test documentation. However, TheIEEE standard, known as 829, specifies in detail a standard for software test documentation.

BS7925-2 shows a diagram of a suggested hierarchy of test documentation.

HOME WORK

Exercise

Putting aside management problems. Read through test documentation examples in BS7925-2 and answer following questions:

What test techniques does component test strategy stipulate?

What percentage of decision coverage is required?

What should be done if errors are found?

The project component test plan is useful because the approach outlined allows:

a) Strict adherence to the component test strategy

b) More faults to be identified in the LOG components

c) A basic working systems to be established as early as possible

d) Isolation of the components within the test strategy

The component test plan must consist of a single document? TRUE/FALSE

The component test plan must specify test completion criteria? TRUE/FALSE

Why does component test plan specify 100% DC whereas strategy required 90%?

Which test case deals with non-numeric input?

List the expected outcome and the test condition

Why does the CTP have additional/altered test cases?

What action has been taken as a result of the test report?

7


8/75

1.5.3 Successful tests detect faults

As the objective of a test should be to detect faults, a successful test is one that does detect afault. This is counter-intuitive, because faults delay progress; a successful test is one thatmay cause delay. The successful test reveals a fault which, if found later, may be many moretimes costly to correct so in the long run, is a good thing.

1.5.4 Meaning of completion or exit criteria

Completion or exit criteria are used to determine when testing (at any stage) is complete.These criteria may be defined in terms of cost, time, faults found or coverage criteria.

1.5.5 Coverage criteria

Coverage criteria are defined in terms of items that are exercised by test suites, such asbranches, user requirements, and most frequently used transactions etc.

1.6 THE PSYCHOLOGYOF TESTING

1.6.1 Purpose

The purpose of this section is to explore differences in perspective between tester anddeveloper (buyer & builder) and explain some of the difficulties management and staff facewhen working together developing and testing computer software.

1.6.2 Different mindsets

We have already discussed that none of the primary purposes of testing is to find faults in

software i.e., it can be perceived as a destructive process. The development process on theother hand is a naturally creative one and experience shows that staff that work indevelopment have different mindsets to that of testers.

We would never argue that one group is intellectually superior to another, merely that theyview systems development from another perspective. A developer is looking to build newand exciting software based on user's requirements and really wants it to work (first time if possible). He or she will work long hours and is usually highly motivated and verydetermined to do a good job.

A tester, however, is concerned that user really does get a system that does what they want, isreliable and doesn't do thing it shouldn't. He or she will also work long hours looking forfaults in software but will often find the job frustrating as their destructive talents take theirtool on the poor developers. At this point, there is often much friction between developer andtester. Developer wants to finish system but tester wants all faults in software fixed beforetheir work is done.

8


9/75

In summary:

Developers:

Are perceived as very creative - they write code without which there would be nosystem! .

Are often highly valued within an organization.

Are sent on relevant industry training courses to gain recognized qualifications.

Are rarely good communicators (sorry guys)!

Can often specialize in just one or two skills (e.g. VB, C++, JAVA, SQL).

Testers:

Are perceived as destructive - only happy when they are finding faults!

Are often not valued within the organization.

Usually do not have any industry recognized qualifications, until now

Usually require good communication skills, tack & diplomacy.

Normally need to be multi-talented (technical, testing, team skills).

1.6.3 Communication b/w developer and tester

It is vitally important that tester can explain and report fault to developer in professionalmanner to ensure fault gets fixed. Tester must not antagonize developer. Tact and diplomacyare essential, even if you've been up all night trying to test the wretched software.

1.6.4 How not to approach

Tester: "Hey Fred. Here's a fault report AR123. Look at this code. Who wrote this? Was ityou? Why, you couldn't program your way out of a paper bag. We really want this fixed by 5

o'clock or else."

We were unable to print Fred's reply because of the language! Needless to say Fred did notfix the fault as requested.

9


10/75

Exercise

Your trainer will split you into small test teams. One of you will be the test team leader. Youhave found several faults in a program and the team leader must report these to the developer(your trainer). The background is that your team has tested this program twice before andtheir are still quite a lot of serious faults in the code. There are also several spelling mistakesand wrong colors on the screen layout. The test team is getting a bit fed up. However, you

have to be as nice as possible to the developer.

1.6.6 Why can't we test our own work?

This seems to be a human problem in general not specifically related to softwaredevelopment. We find it difficult to spot errors in our own work products. Some of thereasons for this are:

We make assumptions

We are emotionally attached to the product (it's our baby and there's nothingwrong with it).

We are so familiar with the product we cannot easily see the obvious faults. We're humans.

We see exactly what we want to see.

We have a vested interest in passing the product as ok and not finding faults.

Generally it is thought that objective independent testing is more effective. There are severallevels of independence as follows:

Test cases are designed by the person(s) writing the software.

Test cases are designed by another person(s).

Test cases are designed by a person(s) from a different section.

Test cases are designed by a person(s) from a different organization.

Test cases are not chosen by a person.

The discussion of independence test groups and outsourcing is left to another section.

10


11/75

1. 7 RE-TESTINGAND REGRESSION TESTING

We find and report a fault in LOG 3, which is duly fixed by the developer and included in thelatest release which we now have available for testing. What should we do now?

Examples of regression tests not carried out include:

The day the phones stopped. . LAS failure on 4th November (perhaps)

Ariane 5 failure.

Whenever a fault is detected and fixed then the software should be re-tested to ensure that theoriginal fault has bee successfully removed. You should also consider testing for similar andrelated faults. This is made easier if your tests are designed to be repeatable, whether they aremanual or automated.

Regression testing attempts to verify that modifications have not caused unintended adverseside effects in the unchanged software (regression faults) and that the modified system still

meets requirements. It is performed whenever the software, or its environment, is changed.

Most companies will build up a regression test suite or regression test pack over time andwill add new tests, delete unwanted test and maintain tests as the system evolves. When amajor software modification is made then the entire regression pack is likely to be run (albeitwith some modification). For minor planned changes or emergency fixes then during the testplanning phase the test manager must be selective and identify how many of the regressiontests should be attempted. In order to react quickly to an emergency fix the test manager maycreate a subset of regression test pack for immediate execution in such situations.

Regression tests are often good candidates for automation provided you have designed and

developed automated scripts properly (see automation section).

In order to have an effective regression test suite then good configuration management ofyour test assets is desirable if not essential. You must have version controlof yourtestDocumentation (testplans, scripts etc.) as well as your test data and baseline databases. Aninventory of your test environment (hardware configuration, operating system version etc.) isalso necessary.

11


12/75

1.8 EXPECTED RESULTS

The specification of expected results in advance of test execution is perhaps one of the mostfundamental principles of testing computer software. If this step is omitted then humansubconscious desire for tests to pass will be overwhelming and tester may perhaps interpret aplausible, yet erroneous result, as correct outcome. .

As you will see when designing test using black box and white box techniques there is ampleroom within the test specification to write down you expected results and therefore no realexcuse for not doing it. If you are unable to determine expected results for a particular testthat you had in mind then it its not a good test as you will not be able to (a) determinewhether it has passed or not and (b) you will never be able to repeat it.

Even with a quick and dirty ad-hoc test it is advisable to write down beforehand what youexpect to happen. This may all sound pretty obvious but many test efforts have floundered byignoring this basic principle.

" The major difference between a thing that might go wrong and a thing that cannot possibly

go wrong is that when a thing that cannot possibly go wrong does go wrong it usually turnsout to be impossible to get at or repair."

--Douglas Adams

12


13/75

SOME TESTING TERMINOLOGY

Faults - a mistake in the code that causes the software to not behave as expected (causes)

Failures - the act of a product not behaving as expected - the manifestation of a fault(symptoms)

Validation - establishing the correspondence between the software and its specification -"are we building the right product?"

Test care - the collection of inputs, predicted results and execution conditions for a singletest

Ad-lib/ad-hoc test care - a test executed without prior planning - especially if the expectedbehaviors is not known prior to running the test.

Pass/fail criteria - decision rules used to determine whether a product passes or fails a giventest

Coincidental correctness - when behavior appears to be what is expected, but it is just acoincidence

Test suite - a collection of test cases necessary to "adequately" test a product

Test plan - a document describing the scope, approach, resources and schedule of intendedtesting activity - identifies features to be tested, the testing tasks, who will do each task, andany risks requiring contingency planning.

Oracle - a procedure, process or magical phenomenon that can determine if the actual

behavior matches the expected behavior

Incident - when a test produces an unexpected outcome, - further effort is necessary toclassify the incident as a software error, a design error, a specification error, a testing error,etc.

Bug report - a method of transmitting the occurrence of a discrepancy between actual andexpected output to someone who cares for "follow-up" - also known as discrepancy report,defect report, problem report, etc.

Work-around - a procedure by which an error in the product can be "by-passed" and the

desired function achieved.

13


14/75

Why Write Programs?

Concept: If you want your computer to do exactly what you want it to do, you must write aprogram.

A computer does nothing on its own. In fact, a computer is a dumb machine with nointelligence whatsoever. Despite what you might read in science fiction stories, a computer

does nothing more than blindly follow instructions supplied by a programmer. Computerscannot think.

Definition: A program is a set of instructions that tells the computer exactly what to do.

When someone buys a computer today, the computer sits on the desk doing nothing until heloads a program into the computer's internal memory and starts running the program. Just asa VCR does not record shows on its own without being programmed to do so, a computerrequires detailed instructions found only in programs.

Suppose that you own rental properties and want to use your computer to track your tenantrecords. Your computer will not help you out in any way until you load and run a rental property program. Where do you find such a program? There are two ways to obtainprograms for computers. You can

(1) Buy one and hope that the program does exactly what you want it to do.

(2) Write your own program.

It's much easier and faster to buy a program that you need. Thousands of programs are on themarket today. In fact, there are so many programs out there that you might not see the needfor writing your own programs.

If you can find a program that does exactly what you want, you are ahead of the game. If youfind a program that meets your exact requirements, you should buy that program becausepurchasing a program is often less expensive and much quicker than writing the sameprogram yourself or hiring programmers to write it for you.

Think about this for a moment, though: If there are so many programs sold today that dovirtually everything, why are programming languages such as Visual Basic continuing tobreak previous sales records each year? The answer is simple: People buy a computer so thatthe computer will do jobs that they need done. Firms cannot adapt their business to acomputer program. They must find programs, or write their own programs, so that the

computer processes information according to the business procedures already in place. Theonly way to ensure that a program exactly fits the needs of a firm is for the firm to develop itsown programs.

14


15/75

Business people are not the only ones who need custom-designed programs. No two peoplemanage their finances exactly the same way; no two scientists need computers for exactly thesame kinds of computations; and no two graphic artists need the same kinds of computerdrawing tools. Although people buy spreadsheets and word processors for their general-purpose computing needs, many people require specialized programs for specific jobs.

The art of programming computers is rewarding not only from a requirements standpoint, but

also on a more personal level. Programming computers is fun! Just as a sculptor looks on afinished work of clay, programmers are often proud of the programs that they write. By thetime you finish this book, you will have written programs that were not available before youwrote them. When you want your computer to do something specific and you cannot find aprogram that does the job exactly the way you want, you will be able to design and write theprogram yourself.

Some Programs are Changeable: There is a third method for getting exactly the program thatyou need if you want to computerize your company's accounting records. Accountingsoftware firms often sell not only accounting programs but also the source code for thoseprograms. The source code is a listing of the program's instructions. By having access to thesource code, you can take what the software company wrote and modify the behavior of theprogram to suit your own requirements.

By starting with a functional program instead of starting from scratch, you save programmingtime and money. Sadly, most non-accounting software firms do not supply the source code.Most programs sold today have been compiled. After compiling, the source code is translatedinto a locked-in executable program. The bottom line is that you cannot easily change thebehavior of compiled programs. For most programs, therefore, you have the choice of buyingthem or writing them yourself from scratch.

Definition: Code is another name for program.

Review: No single program pleases everyone. When a company sells a program, it must begeneral enough to please most purchasers. Some people need programs to behave in aspecific manner in order to fulfill a specific need. They must resort to writing their ownprograms. Luckily, Visual Basic takes a lot of the pain out of writing programs.

15


16/75

A Brief History of Textual Programming

Concept: Computers cannot understand just any language. You must learn a language thatyour computer knows before you can write programs for your computer.

Definition: An application is yet another name for program.

Many people use computers all day long for word processing, database storage, and

spreadsheet analysis, without realizing what is going on behind the scenes. You must alwayskeep in mind that computers cannot think. Your computer does not know how to be a wordprocessor. If you want your computer to do word processing, you must supply detailedinstructions in the form of a program. Only by following the detailed instructions of a wordprocessor program that you load can your computer perform word processing.

It would be nice if writing a program is as easy as telling the computer what you want done.Many people can handle ambiguous instructions, but computers are not smart enough tounderstand vague requirements. Computers can only follow orders given to them, and youmust supply those orders in the form of a program. Therefore, you must supply the programsthat you write. Writing programs, especially complex programs, takes time and severalprocedural steps. Visual Basic speeds the process of creating programs, but even with VisualBasic some programs take time to write and perfect.

Definition: A bug is a program error.

These are the typical steps that most programmers go through when writing programs. First,you have an idea for a program. Next, you use a program-development system, such asVisual Basic, to write the program. Errors, or bugs, often appear in programs because of thedetails needed for even the simplest of programs. Therefore, you must test the programthoroughly and fix the errors. Fixing errors is called debugging. Once all the errors are out ofthe program, you have a finished application.

PROGRAMMING

A computer is an information-processing machine. [Executes given commands.

Uses memory.

Information = data. There are various types of data e.g. numerical, text, graphics orpictures, sound signals etc.

A computer program is a sequence or set of instructions in a programming language.

All data and instructions for a program are stored in the computer's memory in codedform. This coded form is similar to the mores code (-0---0-00-). The coding or

translation is now done automatically. I.e. by commercial or computer programming. Programs will be written in the programming languages of which there are many.

Like Pascal, Cobalt, (used for records), Visual Basic, etc.

16


17/75

For the purpose of this module, Visual Basic (VB) programming language will beused. Hence the VB Integrated Development Environment (IDE) will do thetranslation into coded form, which is a software program.

LOGGING INTO VISUAL BASICVBS(adds topics), enter Open, enterWhen in the form use -Label1. Caption = "Hello Class of 2000"

To delete an object from the form Select Edit menu.

ERRORS IN PROGRAMMING

There are three common errors1Syntax errors: are due to structure or grammar of the language (rules) applied, e.g.more or less spacing, full stops, commas etc.2 Routine errors: are due to non-existing situations like 1 divided by 0, is impossibility.These are errors where the program has instructed the computer to perform an impossibleoperation e.g. as shown above.3Logical errors: are those in the meaning of the program.

To save, always SAVE FORM AS, followed by SAVE PROJECT AS.

The Cost of Bugs

Programming, and testing, to its use by the public, there's the potential for bugs to be found.The Figure below shows how the cost of fixing these grows over time.

Cost

Specification Design Code

17


18/75

Time When Bug Is FoundThe cost to fix bugs increased dramatically over time.

The cost are logarithmic - that is, they increase tenfold as time increases. A bug found andfixed during the early stages when the specification is being written might cost next tonothing or 10 pence in our example. The same bug, if not found until the software is codedand tested, might cost 1 to 10. If a customer finds it, the cost could easily top 100.

WHY DOES SOFTWARE HAVE BUGS?

Miscommunication or no communication - as to specifics of what an applicationshould or shouldn't do (the application's requirements).

Software complexity - the complexity of current software applications can be difficultto comprehend for anyone without experience in modern-day software development.Windows-type interfaces, client-server and distributed applications, datacommunications, enormous relational databases, and sheer size of applications haveall contributed to the exponential growth in software/system complexity. And the useof object-oriented techniques can complicate instead of simplify a project unless it is

well engineered.

Programming errors - programmers, like anyone else, can make mistakes.

Changing requirements - the customer may not understand the effects of changes, ormay understand and request them anyway - redesign, rescheduling of engineers,effects on other projects, work already completed that may have to be redone orthrown out, hardware requirements that may be affected, etc. If there are many minorchanges or any major changes, known and unknown dependencies among parts of the

project are likely to interact and cause problems, and the complexity of keeping trackof changes may result in errors. Enthusiasm of engineering staff may be affected. Insome fast-changing business environments, continuously modified requirements maybe a fact of life. In this case, management must understand the resulting risks, and QAand test engineers must adapt and plan for continuous extensive testing to keep theinevitable bugs from running out of control

Time pressures - scheduling of software projects is difficult at best, often requiring alot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.

Egos - people prefer to say things like: 'no problem', 'piece of cake', 'I can whip thatout in a few hours' 'it should be easy to update that old code'Instead of: 'that adds a lot of complexity and we could end up making a lot ofmistakes' or we have no idea if we can do that; we'll wing it', 'I can't estimate howlong it will take, until I take a close look at it', 'we can't figure out what that oldspaghetti code did in the first place'

If there are too many unrealistic 'no problems', the result is bugs.

18


19/75

Poorly documented code - it's tough to maintain and modify code that is badly writtenor poorly documented; the result is bugs. In many organizations managementprovides no incentive for programmers to document their code or write clear,understandable code. In fact, it's usually the opposite: they get points mostly forquickly turning out code, and there's job security if nobody else can understand it ('ifit was hard to write, it should be hard to read').

Software development tools - visual tools, class libraries, compilers, scripting tools,etc. often introduce their own bugs or are poorly documented, resulting in addedbugs.

Example - Microsoft Word

TEST NO INPUTS EXPECTED RESULTS

I Click on File hold down and Open dialog appears

Drag to Open2 Right click on paragraph Menu options appear - font, paragraph etc.

Cut Copy Paste are grayed out

3 Right click within a table 12 menu options appear related to table

Actions e.g. insert row, delete row

4 Spelling Spelling checking starts and displays firstError but check grammar off(Depends on option)

5 File Print Displays printer dialog

6 Click on print icon Will attempt to print to default printer

Driving License Exercise

Consider the following simple program:

IF age> 16 and age


20/75


1 10

2 16

3 17

4 25

5 26

6 657 70

20


21/75

ATM Exercise

Some simplified rules for an A TM:

a) Card must be valid for this bankb) Correct PIN must be enteredc) Withdrawal must not take account overdrawn unless there is an overdraft agreedd) Notes available are 1 0 and 20 only


1 Valid Card

Incorrect PIN50 requested '"

2 Valid Card

Correct PIN

200 requested

Balance 50No overdraft 'i-

3 Valid Card

Correct PIN

50 requested

Balance 50No overdraft /

4 Valid Card

Correct PIN

50 requested

Balance 400Withdrawn 160 earlier 1-

Telephones Exercise


1 Pick UD receiver

2 Dial 100

3 Dia1142 or 192

4 Dial 999

5 Dial Busy Number

6 Press 5 After HearingEngaged tone

7 Dia11471

8 Dia11571

21


22/75

1. 9 Prioritization of Tests

Since we have discussed already that there is never enough time to do all of the testing thatyou would like an obvious approach is to prioritize the tests This will then mean thatwhenever you stop testing you will have done the best testing possible in the time available.

The test manager should identify the criteria to hi used when prioritizing tests. These willinclude but are not limited to:

Severity.

Probability.

Visibility of failure.

Business priority assigned to the requirement.

Customer wants.

How much change the function has undergone.

How error prone is this module.

How critical is it to the business.

How technically complex is it.

How difficult is it to test.

How costly is it to test.

Keep the test priority with the test at all times. This_ will prevent you from developing anddesigning lo_ priority tests that never get run. Use a scheme of high, medium, low or anumeric scheme or 1,2, 3, 4 but try not to have more than about 4 categories.

NO TEST CASE DESCRIPTION PRI

1 A manual block on a bank is introduced (using the HOLD function)Causing payments to be moved between priority classes by the

Scheduler

2 Failure of the primary scheduler process

3 Internal authentication failures

4 Normal close of day

5 Normal system start up

6 One of each type of request to the scheduler

7 Payments are released correctly from the USER class when limits

Exceeded but funds received from another bank causing position to

Improve

8 Payments can be routed to the scheduler queue from several sourceswith the correct class and user defined priority (including payments

that have been referred or repaired)

9 Payments cancelled manually be central control

22


23/75

10 Regression testing of existing telecommunications links

11 Regression testing of recompiled interface with mainframe system

12 Remote bank changes their limit on us causing payments to beReleased from the USER class (if held due to exceeding limit)

13 Remote bank issues temporary raise

14 Route all payments to the scheduler with parameters set forimmediate release

15 Special processing when internal cut-off time reached

16 Suspend/activate limits

17 Suspend/activate scheduler

1.10 Summary

You should now be familiar with some of the fundamental principles of testing and you willrecognize much of the terminology. The horror stories that you will have heard are not justsensationalist to motivate the course; they are true stories and there are (unfortunately) manymore stories about the costs of not testing systems properly.

In particular you can now:

Recognize and use basic testing terminology.

Understand why testing is necessary.

Define error, fault and failure.

Explain why errors occur.

Give examples of the cost of errors.

Understand why exhaustive testing is unachievable.

Appreciate why testing is a risk management process.

Understand the fundamental test process.

Understand the difference between the mindsets of developers and testers. Understand why you cannot test your own work.

Understand the need for regression testing.

Understand the importance of specifying your expected result in advance.

List some criteria for prioritizing your tests.

23


24/75

Chapter2:

Testing throughout Lifecycle

"What is clear from the Inquiry Team's investigations is that neither the Computer Aided

Dispatch (CAD) system itself, nor its users, were ready for full implementation on 26thOctober 1992. The CAD software was not complete, not properly tuned, and not fully tested.The resilience of the hardware under a full load had not been tested. The fall back option tothe second file server had certainly not been tested."

Extract from the main conclusions of the official report into the failure of the LondonAmbulance Service's Computer Systems on October 26th and 27th 1992.

2.1 OVERVIEW

This module covers all of the different testing activities that take place throughout theproject lifecycle. We introduce various models for testing, discuss the economics oftesting and then describe in detail component, integration, system and acceptancetesting. We conclude with a brief look at maintenance testing.


2.2 OBJECTIVES

Understand the difference between verification and validationtesting activities

Understand what benefits the V model offers over other

models.

Be aware of other models in order to compare and contrast.

Understand the cost of fixing faults increases as you move theproduct towards live use.

Understand what constitutes a master test plan.

Understand the meaning of each testing stage.

24


25/75

2.3 Models for Testing

This section will discuss various models for testing. Definitions of these models willdiffer however the fundamental principles are agreed on by experts and practitioners

alike. We cover verification and validation (V & V), the V-Model and briefly discussa Rapid Application Development (RAD) approach to testing.

2.3.1 Verification and validation (V&V)

Verification is defined by B87925 as the process of evaluating a system or componentto determine whether the products of the given development phase satisfy theconditions imposed at the start of that phase.

Validation is defined by B87925 as determination of the correctness of the productsof software development with respect to the user needs and requirements.

25


26/75

Exercise

Validation and Verification

Complete the definition of testing: ________________________________________

Testing is defined by BS7925 as the process of exercising software to that it satisfiesspecified requirements and to____________________________________________

In simpler terms, verification answers the question "have we built the product right",i.e. is it correct and free from errors, does it meet the specification whereasValidation asks the question" is this the right product?" have the users got what theywanted. To help you remember this use the following:

Verification Is it error free, does it do what was specified?

Validation Is it valid, is this what you really, really want?

2.3.2 V-model

There are many models used to describe the sequence of activities that make aSystems Development Life Cycle (SDLC). SLDC is used to describe activities ofboth development and maintenance work. Three models are worth mentioning.

Sequential (the traditional waterfall model).

Incremental (the function by function incremental model).

Spiral (the incremental, iterative, evolutionary, RAD, prototypemodel).

The three models would all benefit from earlier attention to the testing activity thathas to be done at some time during the SDLC.

Any reasonable model for SDLC must allow for change and spiral approach allowsfor this with emphasis on slowly changing (evolving) design. We have to assumechange is inevitable will have to design for change.

Fact 1.Business is always changing

2.Finding a fault causes change

Result Ease of fixing a fault defines ease of responding to change

Corollary

If we want systems that can be modified and hence maintained,the earlier we start testing and try the change process, the earlierwe will find out how easy it is going to be to maintain the system

26


27/75

2.3.3 Sequential Model

Initial Appraisal

Feasibility Study

Requirements, specification

Functional Specification

Technical Specification

ProgramSpecification

Code Acceptance Test

Analysis

The sequential model often fails to bring satisfactory results because of the late attention tothe testing activity. When earlier phases in the development cycle slip, it is the testing phasethat gets squeezed. This can lead to a limited amount of testing being carried out with theassociated production 'teething' problems.

2.3.4 Plan for wanted waterfall model development of system

The overall project plan for development of a system might be as shown below:

Activities Time

Business study ----Requirements analysis ----User level design ----Technical design ----Program specification ----Creation of code ----Unit testing ----Integration testing -----System testing -----Acceptance testing -----Implementation ------

QC

27


28/75

This is a typical Specify, Design and Build project plan.

All testing and quality control points come late in project and only

done if there is time.

When testing is done so late in project it can reveal costly errors.

Project plan has testing done late because people think that onlyphysical deliverables such as code can be tested. Clearly there has to be abetter way.

The challenge it to devise a better way of developing systems. There isa need to introduce quality control points earlier in the SDLC.

28


29/75

2.3.5 Sequential model plus testing gives 'V' diagram

The V diagram is another way of looking at the sequential development but this timefrom viewpoint of testing activities that need to be completed later in SDLC.

The 'V' diagram in this simple form has been around for a long time and is especiallyuseful as it easily demonstrates how testing work done early in SDLC is used as input

to assurance work later in development.

The V model of SDLC offers considerable benefits over others as it emphasizes building of test data and test scenarios during development and not as an afterthought. The V model also allows for establishment of versions, incrementaldevelopment and regression testing.

Management needs to rename activities, referred to variously as systems testing oracceptance testing. There has always been a phase of development traditionallythought of as the testing phase. This is an historical perception. Testing is not a phasebut rather an activity that must be carried out all through development, giving rise to

the principle of Total Quality.

In the past, system testing was the only type of testing carried out. Testing waschecking that programs, when linked together, met the systems specification.Whether design itself was correct was another matter. The concept of "testing" designbefore programs were coded was given only the most perfunctory attention. This wasfor two reasons:

29


30/75

1. By the time physical design had been done the system was very difficult toalter; modifications caused by design reviews were therefore very unwelcome.

2. The design was documented in terms of physical file layouts and programspecifications, neither of which the user could comprehend. The question ofwhether physical design was correct could be reviewed, but the more importantquestion: "Did the system do what the user wanted?" Was largely neglected.

30


31/75

2.3.6 'V' with test recognized deliverable

The activity of Business Analysis has, as deliverables, the Specification ofRequirements from which Acceptance Test Plan is constructed. To have created, forexample, System Architecture without integration Test Specification is to do onlyhalf the job!

31


32/75

2.3.7 Revised plan for system development

The overall project plan for development of a system might be as shown below. Note the new earlyquality control points.

This plan shows that the creation and running of actual tests are separated. The creation of test

material (acceptance test plans, user system test scripts, technical system tests such as integration,link, recovery restart, etc., and unit test data) is done as the relevant design is done. The potential forautomation is very good and the use of tools to capture the test cases, scripts, etc. will play a big partin making the running tests efficient. The early creation of test material will make the process ofdeveloping a system effective. The emphasis must be on first being effective then being efficient.

32


33/75

2.3.8 Rapid Application Development

The spiral, Rapid Application Development (RAD) model has the benefit of theevolutionary approach. This is an incremental process of build a little then test a little,which has the benefit of attempting to produce a usable but limited version early.

The RAD approach relies upon the quality of the RAD team.

The management issues to address are:

Have I got knowledgeable user input in my team?

Have I got experienced designers and developersin my team?

Am I leaving a good audit trail to support futureMaintenance?

Code

Acceptance Test

UserRequirement

33


34/75

2.4 ECONOMICS OF TESTING

This section looks at some of the economic factors involved in test activities. Although someresearch has been done to put forward the ideas discussed, few organizations have yet toprovide accurate figures to confirm these theories.

Major retailers and car manufacturers often issue product recall notices when they realizethat there is a serious fault in one of their products. Perhaps you can think of other examples.The fixing of the so-called millennium but is probably one of the greatest product recallnotices in history.

Boehm's research suggests that cost of fixing faults increases dramatically as we movesoftware product towards field use. If fault is detected at an early stage of design, it may beonly design documentation that has to change resulting in perhaps just a few hours work.However, as project progresses and other components are built based on faulty design, morework is obviously needed to correct fault once it has been found. This is because designwork, coding and testing will have to be repeated for components of the system that were

previously thought to have been completed.

If faults are found in documentation, then development based on that documentation mightgenerate many related faults, which multiply the effect of the original fault.

Analysis of specifications during test preparation (early test design) often brings faults inspecifications to light. This will also prevent faults from multiplying i.e. if removed earlierthey will not propagate into other design documents.

In summary we suggest that it is generally cost effective to use resources on testingthroughout the project lifecycle starting as soon as possible. The alternative is to potentially

incur much larger costs associated with the effort required to correct and re-test major faults.Remember that the amount of resources allocated to testing is a management decision basedon an assessment of the associated risks. However, few organizations are able to accuratelycompare the relative costs of testing and the costs associated with re-work.

34


35/75

2.5 RELATIVE COSTS TO FIX ERROR

The cost of fixing errors escalates as we move the project towards field use. From an analysis

of sixty-three projects cited in Boehm: Software Engineering Economics (Boehm, 1981).

35


36/75

2.6 High level test planning

You should be aware that many people use the term 'test plan' to describe a documentdetailing individual tests for a component of a system. We are introducing the concept ofhigh level test plans to show that there are a lot more activities involved in effective testingthan just writing test cases.

The IEEE standard for test documentation (IEEE/ ANSI, 1983 [Std 829-1983 D,affectionately known as 829, defines a master validation test plan as follows:

Purpose of master test plan is to prescribe scope, approach, resources and schedule of testing

activities. A master test plan should include the following:

1. Test Plan Identifier2. References3. Introduction4. Test Items

5. Software Risk Issues6. Features to be tested7. Features not to be tested8. Approach9. Item Pass/Fail Criteria10. Suspension Criteria11. Resumption Requirements12. Test Deliverables13. Remaining Testing Tasks14. Environmental Needs15. Staffing and Training Needs

16. Responsibilities17. Schedule18. Planning Risks and Contingencies19. Approvals20. Glossary

36


37/75

2.6.1 SPACE - the final frontier (of testing?)

This may be useful acronym to help you remember what you should include in any high leveltest plan document. The actual format is up to you and will depend upon your own companystandards and other internal considerations. Space stands for Scope, People, Approach,Criteria and Environment and we have tried to map this onto the IEEE 829 headings asfollows:

Scope People Approach Criteria Environment

I.Test plan identifier15. Staffing andtraining

8. Approach9. Itempass/fail

14.Environmental

2.References needs criteria needs3.Introduction

4.Test Items 16. Responsibilities 12. Test deliverables 10.Suspensio

nTest lab

6.Features to be tested13. Remaining testtasks

criteria Test network

7.Features notto be tested

5.Software risk issues 17. Schedule 19. Approvals11.Assumption

Test data

requirements

Use of live data

18. Planning risks and Office requirements Use of testing tools Number of

contingencies cycles es

20.Glossary Motivation/rewards Configurationmanagement

37


38/75

Exercise

High Level Planning

We will use the case study for the London Ambulance Service's new computer aidedDispatch system to discuss the test planning issues. Take a few moments to thinkabout how you would test such a system and we will complete the outline plan as a

group exercise.

Scope

People

Approach

Criteria

Environment

38


39/75

2.7 Component testing

Component testing is described fully in BS-7925 and should be aware that component testingis also known as unit testing, module testing or Program Testing. The definition fromBS7925 is simply the testing of individual software components.

Traditionally, the programmer carries out component testing. This has proved to be lesseffective than if someone else designs and funs the tests for the component.

"Buddy" testing, where two developers test each other's work is more independent and oftenmore effective. However, the component test strategy should describe what level ofindependence is applicable to a particular component.

Usually white box (structural) testing techniques are used to design test cases for componenttests but some black box tests can be effective as well.

We have already covered a generic test process in this course. The component test process isshown in the following diagram:

2.8 INTEGRATION TESTING

Integration is the process of combining components into larger assemblies. From the standardBS-7925 integration testing is defined as "testing performed to expose faults in theinterfaces and in the interaction between integrated components", However, in this sectionwe look at two interpretations of integration testing known as integration testing in the largeand integration testing in the small.

By Integration Testing in the Large we mean testing the integration of the new system orsoftware package with other (complete) systems. This would include the identification of,

39


40/75

and risk associated with, all interfaces to these other systems. Also included is testing of anyinterfaces to external organizations (e.g. EDI - electronic data interchange, Internet) but nottesting of the processing or operation of those external systems.

Exercise

Compare the definition of integration testing in the large with B87925 (50115) definition of

interface testing. Any comments?

We use Integration Testing in the Small in the more traditional sense of integration testingwhere components are assembled into sub-systems and subsystems are linked together toform complete systems. Integration strategies may be incremental or non-incremental andinclude:

bin-ban

Top-down

Bottom-up

Sandwich

Testing approach is directly related to integration strategy chosen.

Stubs & Drivers

A stub is a skeletal or special purpose implementation of a software module, used to developor test a component that calls or is otherwise dependent on it.

A test driver is a program or test tool used to execute software against a test case suite.

TRAINER'S NOTES:

Why not refer to the project EX example in BS7925; in the component test plan for thecomponent LOG3 it has a good diagram showing stubs and drivers.

40


41/75

2.9 SYSTEM TESTING

System testing is defined as the process of testing an integrated system to verify that it meetsspecified requirements.

You will come across two very different types of system testing, Functional system testingand non-functional system testing. In plain English functional system testing is focuses ontesting the system based on what it is supposed to do. Non-functional system testing looks atthose aspects that are important yet not directly related to what functions the systemperforms. For example if the functional requirement is to issue an airline ticket, the non-functional requirement might be to issue it within 30 seconds.

A functional requirement is " a requirement that specifies a function that a system or systemcomponent must perform". Requirements-based testing means that the user requirementsspecification and the system requirements specification (as used for contracts) are used toderive test cases. Business process-based testing based on expected user profiles (e.g.scenarios, use cases).

Non-functional requirements cover the following areas:

1. Load2. Performance3. Stress4. Security5. Usability6. Storage7. Volume8. Install ability9. Documentation10. Recovery

Non-functional requirements are just as important as functional requirement

Exercise

List some non-functional requirements for LAS CAD system.

41


42/75

2.10 ACCEPTANCE TESTING

The definition of acceptance testing in BS7925 states that "acceptance testing is formaltesting conducted to enable a user, customer, or other authorized entity to determine whetherto accept a system or component". Acceptance testing may be the only form of testingconducted by and visible to a customer when applied to a software package. The mostcommon usage of the term relates to the user acceptance testing (VAT) but you should beaware that there are several other uses of acceptance testing, which we briefly describe here.

User acceptance testing - the final stage of validation. Customer should perform or beclosely involved in this. Customers may choose to do any test they wish, normally based ontheir usual business processes. A common approach is to set up a model office where systemsare tested in an environment as close to field use as is achievable.

Contract acceptance testing - a demonstration of the acceptance criteria, which would havebeen defined in the contract, being met.

Alpha & beta testing - In alpha and beta tests, when the software seems stable, people who

represent your market use the product in the same way (s) that they would if they bought thefinished version and give you their comments. Alpha tests are performed at the developer'ssite, while beta tests are performed at the user's sites.

Exercise

What problems would LAS have avoided had they set some acceptance criteria?

2.11 MAINTENANCE TESTING

Maintenance testing is not specifically defined in BS7925 but it is all about testing changesto the software after it has gone into productions.

There are several problems with maintenance testing. If you are testing old code the originalspecifications and design documentation may be poor or in some cases non-existent. Whendefining the scope of the maintenance testing it has to be judged in relation to the changedcode; how much has changed, what areas does it really affect etc. This kind of impactanalysis is difficult and so there is a higher risk when making changes - it is difficult todecide how much regression testing to do.

42


43/75

Chapter 3:

Dynamic Testing Techniques

"The system was not fully tested to a satisfactory level of quality and resiliencebefore full implementation on 26 October 1992."

Extract from the main conclusions of the official report into the failure of the London AmbulanceService's Computer Systems on October 26th and 27th and November 4th 1992.

3.1 OVERVIEW

This module introduces idea of a test case design technique. Broadly, these arecategorized as either functional or structural testing techniques. The advantage ofusing these proven design methods is that they provide a more intelligent andeffective means of identifying tests than a purely intuitive approach. You will beexpected to know two functional techniques and two structural techniques in detail

and be aware there are many other techniques that can be applied to test case design.There are many excellent books on testing techniques some of which are listed in theappendix.

3.2 OBJECTIVES


Understand the difference between black box (functional) and white box(structural) testing techniques.

Be able to name at least three black box techniques.

Understand how to use equivalence partitioning and boundary value analysisto design test cases.

Appreciate the use of state transition testing.

Be able to name at least three white box techniques.

Understand the meaning of statement testing and branch testing.

Be aware that the BS7925 standard details some (but not all) of the recognizedtesting techniques.

Know that all of these testing techniques can be supplemented by errorguessing.

43


44/75

3.3 INTRODUCTION

Each of the techniques we are about to describe has its strengths and weaknesses. Auseful rule of thumb is that if you are having difficulty applying a particular techniqueto a testing problem then perhaps you ought to try a different technique. This section

introduces the different types of testing technique and discusses the differencebetween them.

3.3.1 Functional test techniques

Functional test techniques are often referred to as 'black box' test techniques and the commonparlance is that we are 'doing black box testing'. A functional test technique will help designtest cases based on functionality of component or system under test, without necessarilyhaving to understand underlying detail of software design. Consider functionality of systemof determine test inputs and expected results.

Structural test techniques are sometime called white box text techniques hence term 'whitebox testing'. Glass box testing is less widely used term for structural test case design.Structural test techniques help design test cases based on internal structure and design ofcomponent or system under test. Look at code, database spec, data model, etc., to determinetest inputs and expected results.

Exercise

What's the difference between Black and White box testing?

BASED ON USED BY SUITABLE FOR

Black Box Requirements Independent tester System test

Functionality Users Acceptance test

White Box Internal structure Developer Unit test

Code DB design Designer Link testSub-system test

44


45/75

3.4 BLACK BOX TECHNIQUES

The following list of black box techniques is from BS 7925-2. On this course we willdescribe and give an example of only those ones highlighted in bold:

/

Equivalent Partitioning. Boundary Value Analysis

State Transition Testing

Cause-Effect Graphing /' Syntax Testing /

Equivalence partitioning (EP) is a test case design technique that is based on the premise thatthe inputs and outputs of a component can be partitioned into classes that, according to thecomponent's specification, will be treated similarly by the component.. Thus the result oftesting a single value from an equivalence partition is considered representative of thecomplete partition.

As an example consider any program that accepts days of ht week and months of they year asinputs. Intuitively you would probably not expect to have to test every date of the year. Youwould obviously try months with 30 days (e.g. June) and months with 31 days (e.g. January)and you may even remember to try out the special case of February for both non-leap year(28 days) and leap years (29 days). Equally, looking at the days of the week you would not,depending on the application, test every day. You may test for weekdays (e.g. Tuesday) andweekends (e.g. Sunday). What you are in effect doing is deciding on equivalence classes forthe set of data in question.

Not everyone will necessarily pick the same equivalence classes; there is some subjectivity

involved. But the basic assumption you are making is that anyone value from theequivalence, class, is as good as any other when we come to design the test.

We hope that you can see how this technique can dramatically reduce the number of tests thatyou may have for a particular software component.

45


46/75

Exercise

Consider a software component called GENERATE-GRADING, which is used by the systemto calculate student's grade based on an examination mark and a coursework mark. Thecomponent is passed an exam mark (out of75) and a coursework mark (out of 25) from which

it generates a grade fro the course in the range' A' to 'D'. The grade is calculated from theoverall mark, which is calculated as the sum of the exam mark and coursework marks, asfollows.

OVERALL MARK GRADE

Greater than or equal to 70 A

Greater than or equal to 50 but less than 70 B

Greater than or equal to 30 but less than 50 C

Less than 30 D

Exercise

Identify valid partitions:_________

Identify invalid partitions:________

Identify output partitions:_________

Now derive some test cases based on input exam mark partition:

TEST CASE 1 2 3

Input (exam mark)

Input (coursework)

Total mark (calculated)

Partition tested

Expected output

46


47/75

The standard goes on to derive test cases form the other partitions but in this course we willnot have time to do this completely.

Boundary Value Analysis is base on the following premise. Firstly, the inputs and outputs

of a component can be partitioned into classes that, according to the component'sspecification, will be treated similarly by the component and, secondly, that developers areprone to marking errors in their treatment of the boundaries of these classes. Thus test casesare generated to exercise these boundaries.

Exercise

Look at the boundaries for input exam mark. Choose values on each boundary and one either

side of it in order to generate a set of test cases.

TEST CASE 1 2 3 4 5 6

Input (exam mark)

Input (coursework)

Boundary tested

Expected output

Again the standard continues to describe boundary tests for all of the other valid boundaries.

47


48/75

3.5 White Box Techniques

The following list of white box techniques is from BS79252. On this course we will describeand give an example of only those ones highlighted in bold.

Statement testing

Branch Decision Testing.

Data Flow Testing.

Branch Condition Testing. .

Branch Condition Combination Testing

Modified Condition Decision Testing. LCSAJ Testing.

Random Testing.

Statement testing is a structural technique based on decomposition of a software componentinto constituent statements. Statements are identified as executable or non-executable. Foreach test case, you must specify inputs to component, identification of statement(s) to beexecuted and expected outcome of test case.

Example of program with 4 statements:

X=INPUT;Y=4;

IF X>Y THENMessage= "Limited exceeded"

ELSE

Message= "No problems reported" END IF

z=o

Branch testing requires model of source code, which identifies decisions and decisionoutcomes. A decision is an executable statement, which may transfer control to anotherstatement depending upon logic of decision statement. Typical decisions are found in loops

and selections. Each possible transfer of control is a decision outcome. F or each test caseyou must specify inputs to component, identification of decision outcomes to be executed bytest case and expected outcome of test case.

48


49/75

3.5.1 Comparison of white box techniques from "Software Testing in the Real World"

Each column in this figure represents a distinct method of white-box testing, and each row(1-4) defines a different test characteristics. For a given method (column), "Y" in a given rowmeans that test characteristic is required for method. "N" signifies no requirement. "Implicit"means test characteristic is achieved implicitly by other requirements of method. (@ 1993,1994 Software Development Technologies) reproduced with permission.

3.7Summary

In module three you have learnt that applying a formal, recognized testing technique in asystematic way is the most effective approach to finding errors in software. In particular youcan now

Explain the difference between black box (functional) and white box(structural) testing techniques.

Name at least three black box techniques.

Use equivalence partitioning and boundary value analysis to designtest cases.

Recognize a state transition testing technique.

Name at least three white box techniques.

Understand what the meaning of statement testing and branch testing.

Use the standard BS7925 to find out more about testing techniques.

Know when to apply error-guessing techniques to supplement the

49


50/75

formal techniques.

Chapter 4:

Static testing

"On November 4th the system did fail. This was caused by a minor programmingerror that caused the system to "crash", the automatic change over to the back upsystem had not been adequately tested, and thus the whole system was brought

down".

Extract from the main conclusions of the official report into the failure of the London Ambulance Service 's

Computer Systems on October 26th and 27th and November 4 th 1992.

4.0 STATIC TESTING

4.1 Overview

Static testing techniques is used to find errors before software is actually executedand contrasts therefore with dynamic testing techniques that are applied to a working

system. The earlier we catch an error, the cheaper it is, usually, to correct. Thismodule looks at a variety of different static testing techniques. Some are applied todocumentation (e.g. walkthroughs, reviews and Inspections) and some are used toanalyze the physical code (e.g. compilers, data flow analyzers). This is a huge subjectand we can only hope to give an introduction in this module. You will be expected toappreciate the difference between the various review techniques and you will need tobe aware of how and when static analysis tools are used.

4.2 Objectives


Understand the various review techniques that you can apply todocumentation and code.

Appreciate the difference between walkthroughs, formal reviews andinspections.

Understand how static analysis techniques can detect errors in code.

Understand two complexity metrics (lines ofcode and McCabe's metric).

50


51/75

4.3 REVIEWS AND THE TEST PROCESS

4.3.1 What is a review?

A review is a fundamental technique that must be used throughout the development

lifecycle. Basically a review is any of a variety of activities involving evaluation of

technical matter by a group of people working together. The objective of any review

is to obtain reliable information, usually as to status and/or work quality.

4.3.2 Some history of reviews

During any project, management requires a means of assessing a measuring progress. The so-called progress review evolved as means of achieving this. However, results of those earlyreviews proved to be bitter experiences for many project managers. Just how long can aproject remain at 90% complete? They found that they could not measure 'true' progress untilthey had a means of gauging the quality of the work performed. Thus the concept of thetechnical review emerged to examine the quality of the work and provide input to theprogress reviews.

4.3.3 What can be reviewed?

There are many different types of reviews throughout the development life cycle.

Virtually any work produced during development can be (and is) reviewed. This

includes, requirements documents, designs, database specifications, designs, data

models, code, test plans, test scripts, test documentation and so on.

4.3.4 What has this got to do with testing?

The old fashioned view that reviews and testing are totally different things stemsfrom the fact that testing used to be tacked onto the end of the development lifecycle.However as we all now view testing as a continuous activity that must be started asearly as possible you can begin to appreciate the benefits of reviews. Reviews are theonly testing technique that is available to us in the early stages of testing. At earlystages in the development lifecycle we obviously cannot use dynamic testingtechniques since the software is simply not ready for testing.

Reviews share similarities to test activities in that they must be planned (what are wetesting), what are the criteria for success (expected results) and who will do the work

(responsibilities). The next section examines the different types of review techniquesin more detail.

51


52/75

4.4 TYPES OF REVIEW

Walk-through, informal reviews, technical reviews and Inspections are fundamentaltechniques that must be used throughout the development process. All have theirstrengths and weaknesses and their place in the project development cycle. All fourtechniques have some ground rules in common as follows:

A structured approach must be for the review process.

Be sure to know what is being reviewed - each component must have a uniqueidentifier.

Changes must be configuration controlled.

Reviewers must prepare.

Reviewers must concentrate on their own specialization.

Be sure to review the product, not the person.

There must be:

Total group responsibility.

Correct size of review group.

Correct allocation of time.

Correct application of standards.

Checklists must be used.Reports must be produced.Quality must be specified in terms of:

Adherence to standards.

Reliability required.

Robustness required.

Modularity.

Accuracy.

Ability to handle errors.

52


53/75

4.5 REVIEW TEAM SIZE AND COMPOSITION

Problems with small teams must be avoided; bring in extra people, (perhaps use theTesting Team) to bring extra minds to bear on the issues.

Opinion is often divided as to whether or not the author should participate in a review. Thereare advantages to both scenarios. Since specifications and designs must be capable of beingunderstood without the author present, an Inspection without them tests the document.Another reason for excluding the author from the review is that there should be teamownership of the product and team responsibility for the quality of all the deliverables,maintaining ownership via the author runs against this.

Alternatively, including the author can be a valuable aid to team building. Equally, an authormay well be able to clear up misunderstandings in a document more quickly than anotherteam member, thus saving the reviewer valuable time. From a practical perspective however,it is worth remembering that an author is the least likely person to identify faults in thedocument.

The one person who should not attend reviews is the manager. If, as in some cases, themanager is also a contributor to the deliverables, he should be included but treated the sameas other group members. It is important that reviewers are a peer group process.

4.6 TYPE 1, 2 AND 3 REVIEW PROCESSES

Review process is both most effective and universal test method and managementneeds to make sure that review process is working as effectively as possible. Usefulmodel for manager is 1,2,3 model.

1, 2, 3 model is derived from work of first working party of British Computer Society

Specialist Interest Group in Software Testing and book that came from this work; inSoftware Development.

Type 1 testing is process of making sure that product (document, program, screendesign, clerical procedure or Functional Specification) is built to standards andcontains those features that we would expect from the name of that product. It is a testto make sure that produce conforms to standards, is internally consistent, accurate andunambiguous.

Type 2 testing is process of testing to see if product conforms to requirements as specified byoutput of preceding project stage and ultimately Specifications of Requirements for whole

project. Type 2 testing is backward looking and is checking that product is consistent withpreceding documentation (including information on change).

Type 3 testing is forward looking and is about specification of certification processand test that are to be done on delivered product. It is asking question - Can we canbuild deliverables (test material, training material, next stage analysisdocumentation)?

53


54/75

4.6.1 Make reviews incremental

Whilst use of 1, 2, 3 model will improve review technique, reviewing task can bemade easier by having incremental reviews throughout product construction.

This will enable reviewers to have more in-depth understanding of product that theyare reviewing and to start construction type 3 material.

4.6.2 General review procedure for documents

Test team will need general procedure for reviewing documents, as this will probably

form large part of team's work.

1. Establishing standards and format for document. 2. Check contents list.3. Check appendix list.4. Follow up references outside documents.5. Cross check references inside document.6. Check for correct and required outputs.7. Review each screen layout against appropriate standard, data dictionary, processingrules and files/data base access.8. Review each report layout against appropriate standard, data dictionary, processingrules and files/data base access.

9. Review comments and reports on work andreviews done prior to this review.10. Documents reviewed will range from whole reports such as Specification ofRequirements to pages from output that system is to produce. All documents willneed careful scrutiny.

4.6.3 Report on review

Report should categorize products:

Type1 Type 2

Type 3

54


55/75

Defective All agreed Total rework

55


56/75

Defective but All agreed Rework andsoluble solution review all

acceptable material

Possible Some but not all Seek explanationdefect, needs agree possibly review

explanation some of work

Quality issue Prefer an Costs comparedalternative to standards

Acceptable All accept Proceed

Over Most agree Proceed butdeveloped for review budgetsthe budget

Be sensitive to voice of concerned but not necessarily assertive tester in team; thisperson may well have observed a fault that all others have missed. It should not bethat person with load voice or strong personality is allowed to dominate.

4.7 INSPECTIONS

This section on Inspections is based on an edited version of selected extracts fromTom Gib and Dorothy Graham's book [Gib, Graham 93].

4.7.1 Introduction

Michael E. Fagan at IBM Kingston NY Laboratories developed the Inspectiontechnique. Fagan, a certified quality control engineer, was a student of the methods ofthe quality gurus W. Edwards Deming and J. M. Juran.

Fagan decided, on his own initiative, to use industrial hardware statistical qualitymethods on a software project he was managing in 1972-74. The project consisted ofthe translation of IBMs software from Assembler Language to PLS, a high-level programming language assembler. Fagan's achievement was to make statisticalquality and process control methods work on 'ideas on paper'. In 1976 he reported hisresults outside IBM in a now famous paper [Fagan, 1976].

The Inspection technique was built further by Caroline L. Jones and Robert Mays atIBM [jones, 1985] who created a number of useful enhancements:

The kickoff meeting, for training, goal setting, and setting a strategy for thecurrent inspection

Inspection cycle;

The causal analysis meeting;

The action database;

The action team.

56


57/75

4.7.2 Reviews and walk-through

Reviews and walkthroughs are typically peer group discussion activities - withoutmuch focus on fault identification and correction, as we have seen. They are usuallywithout the statistical quality improvement, which is an essential part of Inspection.Walkthroughs are generally a training process, and focus on learning about a singledocument. Reviews focus more on consensus and buy-in to a particular document.

It may be wasteful to do walkthroughs or consensus reviews unless a