istqb tfc v2.3

• EDS Internal

03-23-05Version 2.3

EDS ISTQB Testing Foundation CourseEDS ISTQB Testing Foundation Course

Presentation by Paul Weymouth, UK Testing ADU

• EDS InternalEDS ISTQB Testing Foundation Course Version 2.3

How to Navigate the CourseHow to Navigate the Course

• Click on the symbol to go to the next slide.

• Click on the symbol to go to the previous slide.

• Click on the symbol to go to the main Areas Covered slide.

• Click on the symbol to move on to associated slides.

• Click on the symbol to go back to the previous menu.

• You may also see the following symbols displayed :-

EDS specific information available – used where an EDS exceptions exist. EDS

If this appears top right, it means animation applies to the slide. It will also appear

bottom right after the final animation mouse click. For Tutor’s use only.


Other SymbolsOther Symbols

Refers to the ISTQB Glossary Of Terms.

You will see this symbol appear alongside the definition of a term found in this Glossary

These key terms are important to learn.

This is a ‘Pearl of Wisdom’. Typically a quote from an external testing guru, that helps to re-enforce what we have just learned. You’ll find a few of these dotted around the course.


ObjectivesObjectives

• The primary objective of this course is to provide testing foundation level training to staff involved in testing related activities (I.e. developers, testers, managers)

• It is anticipated that after taking this course staff will be in a position to take and pass the external ISTQB Foundation exam in Software Testing


Areas coveredAreas covered

Fundamentals of testing

Testing Throughout the Software Lifecycle

Static Techniques

Test Design Techniques

Test Management

Tool Support for Testing

Index of key terms

• EDS Internal

03-23-05Version 2.3

Fundamentals of TestingFundamentals of Testing


Fundamentals of TestingFundamentals of Testing

Why Testing is Necessary

What is Testing

General Principles of Testing

Fundamental Test Process

The Psychology of Testing

Summary

Areas Covered


Why Testing is NecessaryWhy Testing is Necessary

• Software Systems are now part of our everyday life

• They are used almost everywhere, for example in:

- Banking and Financial institutions

- Retail industry

- Central and Local Government

- Transport (e.g. Planes, Trains and Automobiles)

- Medicine (Hospitals, research centres)

- Home Entertainment

• We have all experienced Software Systems failing!

Software Systems – Some context



Software System Failures can lead to:

– Human Injury or Death

• e.g. airplanes crashing

– Technological disasters

• e.g. Missile Systems malfunctioning

– Legal action and associated costs

• e.g. failure to meet contractual obligations

– Loss of face for suppliers and/or their customers;

• e.g. mis-spelling company name on mail shots

Software Systems – When things go wrong



• A Human can make an Error (aka a Mistake)

• An Error is ‘A Human Action that produces an Incorrect Result’

• The Error can cause a Defect (aka a Fault or Bug)

• A Defect is ‘A flaw in a component or system that can cause the component or system to fail to perform its required function’

• A Defect can be in the Software, System or in a Document

* ISTQB Standard Glossary of Terms

Causes of Software Failure



• Defects occur because human beings are fallible

• Also because of:

– time pressure

– complex code

– complex infrastructure

– changed technologies

– and/or many system interactions

Errors, Defects and Failures



• A Defect may result in a Failure

• A Failure is a ‘Deviation of the component or system from its expected delivery, service or result’

• Failures can be caused by environmental conditions as well

– E.g. radiation, magnetism, electronic fields

– Pollution can cause faults in firmware or influence the execution of software by changing hardware conditions.


* ISTQB Standard Glossary of Terms




Error

Defect

Failure

a human action that produces an incorrect result

A flaw in a component or system that can cause the

component or system to fail to perform its required function

Deviation of the component or system from its expected delivery, service or result

May result in

Can manifest as



• Rigorous testing of systems and documentation can:

– reduce the risk of problems occurring in an operational environment

– contribute to the quality of the software system

• How? By finding and correcting defects before the system is released for operational use

• Software testing may also be required to meet contractual or legal requirements, or industry-specific standards

The Role of Testing



• Testing can help us measure the Quality of software

• Quality - ‘The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations’

• This is measured in terms of defects found

• Defects covering:

– functional software requirements and characteristics

– and non-functional software requirements and characteristics (e.g. reliability, usability, efficiency, portability and maintainability)

• Testing can give confidence in the Quality of the software if it finds few or no defects

Testing and Quality


Why Testing is Necessary Why Testing is Necessary

• A properly designed test that passes, reduces the overall level of Risk in a system

• Risk – ‘A factor that could result in future negative consequences; usually expressed as impact and likelihood’

• When testing does find defects, the Quality of the software system increases when those defects are fixed

• The Quality of systems can be improved through Lessons learned from previous projects

• Analysis of root causes of defects found in other projects can lead to Process Improvement

• Process Improvement can prevent those defects reoccurring

• Which in turn, can improve the Quality of future systems

• Testing should be integrated as one of the Quality assurance activities

Testing and Quality


Testing TerminologyTesting Terminology

• “Quality is not intangible.

The purpose of testing is to make quality visible.

Testing is the measurement of software quality!"

Bill Hetzel 1988

Testing Pearl of Wisdom



Why exhaustive testing is impossible

• Exhaustive testing of complex software applications: – requires enormous resources– is too expensive– takes too long

• It is therefore impractical

• Need an alternative that is pragmatic, affordable, timely and provides results



• “In any form of testing it is impossible to achieve total confidence.

The only exhaustive testing there is ………

is so much testing, that the tester is exhausted!"

Bill Hetzel 1988



Why Testing is Necessary Why Testing is Necessary

System has 20 screensAverage 4 menus / screenAverage 3 options / menuAverage of 10 fields / screen2 types of input per fieldAround 100 possible values

Approximate total for exhaustive testing20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests

Test length = 1 sec then test duration = 17.7 daysTest length = 10 sec then test duration = 34 weeksTest length = 1 min then test duration = 4 yearsTest length = 10 mins then test duration = 40 years!

Why don’t we test everything ?



• Deciding how much testing is enough should take account of:

– the level of Risk

– project constraints such as time and budget

• Risks should be evaluated at the Business Level, Technological Level, Project Level and Testing Level

• Risks are also used to decide where to start testing and where more testing is needed

• Risk considerations can include:

– financial implication of software being released that is un-tested (support costs / possible legal action)

– software being delivered late to market

– potential loss of Life (safety critical systems)

– potential loss of face (may have financial implications as well)

So, How Much Testing is Enough?



• Risk analysis should be used to determine what to test in each component and just as importantly what not to test

• For example, an unacceptable risk would say we must test, an acceptable one perhaps not to test

• Testing is a risk-control activity that provides feedback to the stakeholders

• With this feedback the stakeholders can make informed decisions about the release of the software (or system) being tested

• More about Risks later in the Course



Why Testing Is NecessaryWhy Testing Is Necessary

• Exit criteria is used to determine when testing at any stage is complete The set of generic and specific conditions, agreed upon with the stakeholders, for permitting a process to be officially completed

• Exit criteria may be defined in terms of :-

– Thoroughness – i.e. coverage or requirements

– cost or time constraints

– percentage of tests run without incident

– number of faults remaining



What is Testing?What is Testing?

• When asked, people often think that Testing only consists of running tests, i.e. executing the software

• Testing – ‘The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.’

• Test execution is only a part of testing, but not all of the testing activities

• Test activities exist before and after test execution

A Definition (and a Misconception)



• Activities such as:

– Planning and control

– Choosing test conditions

– Designing test cases

– Checking results

– Evaluating completion criteria

– Reporting on the testing process and system under test

– Finalizing or closure (e.g. after a test phase has been completed)

• Testing also includes reviewing of documents (including source code) and static analysis

A Definition (and a Misconception)


Why Testing is necessaryWhy Testing is necessary

• “Any activity that is undertaken with the objective of helping us to evaluate or measure an attribute of our software should be considered a testing activity”

Hetzel 1998




• There are different test objectives:

– To find defects

– To gain confidence about the level of quality and to provide information

– To prevent defects

• Both dynamic testing and static testing can be used as a means for achieving these objectives

• They provide information in order to improve:

– The system to be tested

– The development and testing processes

– Live operations (e.g. how long it takes for a process to run)

Test Objectives



• By designing tests early in the project life cycle it can help to prevent defects from being introduced into code

• Reviews of documents throughout the lifecycle (e.g. requirements and design) also help to prevent defects appearing in the code. More about this when we cover Static techniques

Test Objectives


0102030405060708090

100

Reqs Des Code Unit Accept Use

Cost

Test Objectives – cost of fault correction


Relative

Multiples



• The Objectives of testing can vary depending on the stage of testing being conducted. E.g.:

– Development testing (e.g. component, integration and system testing) - to cause as many failures as possible so that defects in the software are identified and can be fixed

– Acceptance testing - to confirm that the system works as expected, to gain confidence that it has met the requirements

– Maintenance testing - often includes testing that no new errors have been introduced during development of the changes

– During Operational testing - may be to assess system characteristics such as reliability or availability

Test Objectives



• In some cases the main Objective of testing may be to assess the quality of the software (with no intention of fixing defects), to give information to stakeholders of the risk of releasing the system at a given time

• More on these test stages later in the course

Test Objectives



• “Testing is the process of executing a program or system with the intent of finding errors”

Myers 1979

• "Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results"

Hetzel 1988




• Debugging and testing are different:

– Testing can show failures that are caused by defects

– Debugging identifies the cause of a defect, repairs the code and checks that the defect has been fixed correctly

• Testing then ensures that the fix does indeed resolve the failure

• The responsibility for each activity is very different, i.e.

– Testers test

– Developers debug

Testing v Debugging


General Testing PrinciplesGeneral Testing Principles

1. Testing shows presence of Defects

2. Exhaustive Testing is Impossible!

3. Early Testing

4. Defect Clustering

5. The Pesticide Paradox

6. Testing is Context Dependent

7. Absence of Errors Fallacy

The Seven Key Principles



1. Testing shows the presence of Defects

• We test to find Faults (a.k.a Defects)

• As we find more defects, the probability of undiscovered defects remaining in a system reduces.

• However Testing cannot prove that there are no defects present



Why Testing is necessaryWhy Testing is necessary

• “The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that program”

• “Do not plan a test effort under the tacit assumption that no errors will be found”

• “A good test is one that has a high probability of detecting an as yet undiscovered error”

• “A successful test is one that detects an as-yet undiscovered error”

Myers 2004

Testing Pearls of Wisdom



2. Exhaustive Testing is Impossible!

• We have learned that we cannot test everything (i.e. all combinations of inputs and pre-conditions).

• That is we must Prioritise our testing effort using a Risk Based Approach.




3. Early testing

• Testing activities should start as early as possible in the development life cycle

• These activities should be focused on defined objectives – outlined in the Test Strategy

• Remember from our Definition of Testing, that Testing doesn’t start once the code has been written!




4. Defect Clustering

• Defects are not evenly spread in a system

• They are ‘clustered’

• In other words, most defects found during testing are usually confined to a small number of modules

• Similarly, most operational failures of a system are usually confined to a small number of modules

• An important consideration in test prioritisation!




5. The Pesticide Paradox

• Testing identifies bugs, and programmers respond to fix them

• As bugs are eliminated by the programmers, the software improves

• As software improves the effectiveness of previous tests erodes

• Therefore we must learn, create and use new tests based on new techniques to catch new bugs

• N.B It's called the "pesticide paradox" after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether different approach.’ – Beizer 1995




6. Testing is Context Dependent

• Testing is done differently in different contexts

• For example, safety-critical software is tested differently from an e-commerce site

• Whilst, Testing can be 50% of development costs, in NASA's Apollo program it was 80% testing

• 3 to 10 failures per thousand lines of code (KLOC) typical for commercial software

• 1 to 3 failures per KLOC typical for industrial software

• 0.01 failures per KLOC for NASA Shuttle code!

• Also different industries impose different testing standards




7. Absence of Errors Fallacy

• If we build a system and, in doing so, find and fix defects ....

It doesn’t make it a good system

• Even after defects have been resolved it may still be unusable and/or does not fulfil the users’ needs and expectations



Fundamental Test ProcessFundamental Test Process

The five stages of the fundamental test process

• Test Planning and Control

• Test Analysis and Design

• Test Implementation and Execution

• Evaluating Exit Criteria and Reporting

• Test Closure Activities



• The process always starts with planning and ends with test closure activities

• Each phase may have to be executed a number of times in order to fulfil exit or completion criteria

• Although logically sequential, the activities in the process may overlap or take place concurrently

Fundamental Test Process



Test Planning

• Specifies how the test strategy and project Test Plan

A document describing the scope, approach, resources and schedule of intended test activities

apply to the software under test

• Principally:

– verify the mission

– define the Test objectives

– Specify the Test Activities required to meet the mission and objectives

Test Planning and Control



Test Planning (continued)

• Major Tasks are :-

– Identify the objectives of testing

– Determine Scope

– Determine the Test Approach

– Determine the required test resources

– Implement the test policy and/or the test strategy

– Schedule test analysis and design tasks

– Schedule test implementation, execution and evaluation

– Determine the Exit Criteria

• More on Test Planning in Test Management section




Test Control

• The ongoing activity of comparing actual progress against the plan

• Reporting status, including deviations from the plan

• Taking actions necessary to meet the mission and objectives of the project

• Test Planning takes into account the feedback from monitoring and control activities.

• Major Tasks are :-

– measure and analyse results

– Monitor and document progress, test coverage and exit criteria

– initiate corrective actions

– make decisions

• More on Test Control in Test Management section




• General testing objectives are transformed into tangible Test Conditions (An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element)

and Test Designs (A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases).

• Tests should be designed using the test design techniques selected in the test planning activity

• Major tasks are:– Review the test basis – From Analysis of test items, identify Test Conditions/Requirements and

required Test Data – Design the tests (note – the detail, in the form of a Test Case, is

developed in the next stage)– Evaluate testability of the requirements and system – Design the test environment set-up – Identify any required infrastructure and tools

Test Analysis and Design



• “The act of designing tests is one of the most effective error prevention mechanisms known

... The thought process that must take place to create useful tests can discover and eliminate problems at every stage of development.“

Beizer 1983




• Test Conditions are transformed into Test Cases and Testware

• The test environment is created

• Major tasks are:

Create Test Cases

– Develop and prioritise Test Cases

– Create test data

– Write test procedures

– Preparing test harnesses

– Write automated test scripts

– Create test suites from the test cases for efficient test execution

Check the Environment

– Verify that the test environment has been set up correctly

Test Implementation and Execution



• Each Test Case is specified in terms of :-

– its objective

– the initial state of the system

– the input

– the expected result.




• Major tasks (continued):

Execute the Tests

– Execute the Test Cases according to the planned sequence.

– Log the outcome of test execution

– Test execution records should uniquely identify the versions of the software under test, test tools and Testware

– It should be possible to establish that all testing has been carried out by reference to the test records.




• Major tasks (continued):

Check the Results

– Compare actual results with expected results

– Report discrepancies as Incidents

– Analyse Incidents to establish Root cause

– Repeat, as necessary, test activities as result of action taken for each discrepancy

– The test coverage levels achieved for those measures specified as test completion criteria should be recorded.




•“Thoroughly inspect the results of each test”

Myers - 2004


Ref: Myers, The Art of Software Testing, J Wiley and Sons, 1979



• Test execution is assessed against the objectives defined in Test Planning

• This should be done for each Test Level (i.e. test stage)

A group of test activities that are organized and managed together.


– Check test logs against the exit criteria specified in Test Planning

– If the exit criteria has not been met

• Assess if more tests are needed

• Assess which test activities may need to be repeated

– Assess if the exit criteria specified should be changed

– Produce a test summary report for stakeholders review

Evaluating Exit Criteria and Reporting



• Collect data from completed test activities to consolidate experience, Testware, facts and numbers


– Check which planned deliverables have been delivered

– Check that Incident reports status are up-to-date (e.g. Closed)

– Ensure all Incident reports have associated change records

– Record acceptance of the system

– Finalise and archive Testware, the test environment and the test infrastructure for later reuse

– Handover Testware to the maintenance organization

– Analyse lessons learned for future releases and projects, and for process improvement.

Test Closure Activities


Fix component test plan and repeat

Fix test design and repeat

Fix test design and repeatFix component or test cases/scriptsand repeat

5 Phases of the Fundamental Test Process


Test Planning

and Control

Test Analysis

and Design


Evaluating Exit Criteria and Reporting

Test Closure Activities


PsychologyPsychology of Testingof Testing

• Historically testing was viewed as showing the system meets its requirements

• This has evolved to a stage where testing is performed with the primary aim of finding faults rather than proving correctness. It is perceived as a destructive process

• Seeking to find failures (the right mindset) can be viewed as criticism of the product and/or its author

• But looking for failures is constructive!

– Time can be saved

– Risks reduced

– Costs reduced

– Skills improved

The Testing Mindset



• It is important that the Objectives of testing are clearly understood as humans will moderate their behaviour accordingly (however sub-consciously):-

– “If testing is showing the system meets its requirements then I will just produce tests that show this.”

– “If testing is aimed at finding faults then I will be measured on this so I will put effort into designing tests that are more likely to find faults.”

• The Testing mindset is different from a Developer’s

The Testing Mindset


Psychology of TestingPsychology of Testing

• “Testing is an extremely creative and

intellectually challenging task”

• “Tests must be written for invalid and unexpected, as well as valid and expected, input conditions”

Myers - 1979




• A Tester needs:– good communication skills

– good observation skills

– people handling skills

– Curiosity

– patience

– reliability

– thoroughness

– an inquisitive nature

– attention to detail

– creativity in terms of identifying likely faults

– Experience

• However as with most other disciplines an effective test team will need a mix of skills so it is difficult to generalise

Traits of Good Testers



• The relationship between a Developer and a Tester is not normally an easy one because:-

– testers point out problems with software

– developers like to think their software is perfect

– testers are perceived as delaying the project by finding faults in the system

– when the development slips testers normally have to work long hours to test the product, which in turn can cause resentment.

• It is important that they work together

• It is also important that they have mutual respect for each other.

• Collaboration is the right approach – we work to a common goal!

• Communicate findings objectively, not subjectively

Developer vs Tester Relationship



• The right mindset could enable Developers to test the code

• However, passing this responsibility to trained and professional testing resources has many benefits (such as higher defect find rates)

• Authors tend to bring across assumptions they have made when developing the software. They are less likely to write tests to show faults in their own software (human nature)

• With testing performed by independent testers, testing effort is focused and not compromised by development effort and bias

• It is generally believed that objective independent testing is more effective

Independent testing



• There are several levels of Independence (from Low to High)

– Tests designed by the person(s) who wrote the software under test

– Tests designed by another person(s) (e.g. from the development team).

– Tests designed by a person(s) from a different organizational group (e.g. an independent test team).

– Tests designed by a person(s) from a different organization or company (e.g. outsourcing to an in-house or external test specialist organisation)

Independent testing


Fundamentals of Testing - SummaryFundamentals of Testing - Summary

• We learned of the possible effects of Defects on people, on the environment and on companies

• We learned of the causes of a Defect and the results – Error/Defect/Failure – remember these and alternative terms

• We learned why testing was necessary – how it improves Quality and reduces Risk of failure

• We learned how root cause analysis leads to process improvement

• We learned why you can’t test everything and when to stop testing – through Risk Analysis, Prioritisation and use of Exit Criteria


Fundamentals of Testing - SummaryFundamentals of Testing - Summary• We learned about Testing:

– its main Objectives, i.e. to find and prevent defects and to gain confidence in system Quality

– How we meet these Objectives– How they can vary by Test Level – That it is conducted before and after the code is delivered– What activities it comprises– That Debugging and Testing are different

• We learned of the 7 Key Principles of Testing:

– Testing shows presence of Defects– Exhaustive Testing is Impossible!– Early Testing– Defect Clustering – The Pesticide Paradox– Testing is Context Dependent– Absence of Errors Fallacy


Fundamentals of Testing - SummaryFundamentals of Testing - Summary

• We learned the Fundamental Test process:

– The Stages:

• Test Planning and Control

• Test Analysis and Design

• Test Implementation and Execution

• Evaluating Exit Criteria and Reporting

• Test Closure Activities

– And the main tasks for each stage of the process

• And we learned about the Psychology of Testing:

– What is the right testing mindset (looking for bugs!)– Why it is constructive not destructive!– The skills needed for a good tester– Tester and Developer relationship issues– The benefit and types of Independent testing

• EDS Internal

03-23-05Version 2.3

Testing Throughout the Software Life-cycleTesting Throughout the Software Life-cycle


Testing Throughout the Software LifecycleTesting Throughout the Software Lifecycle

Software Development Models

Test Levels

Test Types – the Targets of Testing

Maintenance Testing

Summary


Software Development ModelsSoftware Development Models

The V-Model

Iterative Development Models

Testing within a Lifecycle Model


V-ModelV-Model

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


V-ModelV-Model

The benefits of the V Model include:

• The testing phases are given the same level of management attention and commitment as the corresponding development phases

• The outputs from the development phases are reviewed by the testing team to ensure their testability

• Verification and validation (and early test design) can be carried out during the development of the software work products

• The early planning and preliminary design of tests provides additional review comments on the outputs from the development phase


V-ModelV-Model

• The levels of development and testing shown in the model vary from project to project

• For example, there may additional test levels, such as System Integration Testing, sitting between System Testing and Acceptance Testing (more on these test levels later)

• The work products coming out from any one development level may be utilised in one or more test levels

• For example, whilst the prime source for Acceptance testing is the Business Requirement, the System Requirements (e.g. Use Cases) may also be needed to support detailed test design


V ModelV Model

• “The V Model provides a powerful tool for managing and controlling risk within the testing component of a project.

The process of bringing testing planning and design into the development process as early as possible enables risks to be identified and strategies for removing or mitigating

them to be put in place in a timely manner.”

Watkins - 2001



Iterative Development ModelsIterative Development Models

• Iterative Development – Establish Requirements– Design the System– Build the System– Test the System

• Achieved with small developments – Iterations and Increment within Iterations

• As Increments are developed and tested the System grows and grows. Need for more testing with Regression Testing paramount

• E.g. RAD, RUP and Agile development models

• Agile development – aim is to deliver software early and often– Rapid production and ‘time to market’– Can handle (and anticipates) changing requirements throughout all development and test phases


Iterative Development ModelsIterative Development Models

• Rapid Application Development

User Requirements

Code

Acceptance Test


Testing within a Lifecycle ModelTesting within a Lifecycle Model

• Characteristics of good testing in any life cycle model:

– A Test Level exists for every development Level– Each Test Level has specific objectives– Test analysis and design for each Test Level begins during corresponding development Level– Early and proactive involvement of testers in reviewing development deliverables – benefits both parties

• Test Levels should be adapted depending on Project nature. May be better to combine Test Levels, e.g. with COTS testing.


Test LevelsTest Levels

Component Testing

Integration Testing

System testing

Acceptance Testing


Component TestingComponent Testing

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan



• Component – A minimal software item that can be tested in isolation.

• Component Testing – The testing of individual software components.

• Sometimes known as Unit Testing, Module Testing or Program Testing

• Component can be tested in isolation – stubs/drivers may be employed

• Test cases derived from component specification (module/program spec)

• Functional and Non-Functional testing

• Usually performed by the developer, with debugging tool

• Quick and informal defect fixing

Definition



• Test-First/Test-Driven approach – create the tests to drive the design and code construction!

• Instead of creating a design to tell you how to structure your code, you create a test that defines how a small part of the system should function.

• Three steps:1. Design test that defines how you think a small part of the software

should behave (Incremental development).

2. Make the test run as easily and quickly as you can. Don't be concerned about the design of code, just get it to work!

3. Clean up the code. Now that the code is working correctly, take a step back and re-factor to remove any duplication or any other problems that were introduced to get the test to run.

Russell Gold, Thomas Hammell and Tom Snyder - 2005

Definition


Integration Testing Integration Testing

Definition

Component Integration Testing

System Integration Testing


Integration Testing Integration Testing

Definition

• Integration Testing - Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems

• Components may be code modules, operating systems, hardware and even complete systems

• There are 2 levels of Integration Testing

– Component Integration Testing

– System Integration Testing


Component Integration TestingComponent Integration Testing

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


Component Integration Testing Component Integration Testing

Definition

• component integration testing Testing performed to expose defects in the interfaces and interaction between integrated components

• performed by the test team

• usually formal (records of test design and execution are kept)• all individual components should be integration tested prior to

system testing



• To consider - should the integration testing approach:

– Start from top level components and work down?– Start from bottom level components and work up?– Use the big bang method?– Be based on functional groups?– Start on critical components first?– Be based on business sequencing? Maybe suit System Test needs.

• Knowledge of the system architecture is important • The greater the scope of the integration approach the more difficult

it is to isolate defects• Non-Functional requirements testing may start here – e.g. early

performance measures

Test Planning



Top-down testing

Stubs

Q R

S T U V

Component under test

P



Top-down testing

Pro’s• provides a limited working system

early in the design process• depth first integration demonstrates

end-to-end functions early in the development process

• early detection of design errors through early implementation of the design structure

• early testing of major control or decision points

Con’s• stubs only provide limited

simulations of lower level components and could influence spurious results

• breadth first means that higher levels of the system must be artificially forced to generate output for test observations



Bottom-up testing

Q R

S T U

P is the driver for components Q and R

V

Component under testP

Same for Q and R driving their components



Bottom-up testing

• using drivers instead of upper level modules to simulate the environment for lower level modules

• necessary for critical, low level system components

• testing can be observed on the components under test from an early stage

• unavailability of a demonstrable system until late in the development process

• late detection of system structure errors

Pro´s Con´s



Big Bang Approach

Main Menu Function 1 Function 2 Function 3

Component 1 Component 2 Component 3 Component 4 Component 5 Component 6

Not usually the preferred approach


Component Integration Testing Component Integration Testing

The following testing techniques are appropriate for Integration Testing:

• Functional Testing using Black Box Testing techniques against the interfacing requirements for the component under test

• Non-functional Testing (where appropriate, for performance or reliability testing of the component interfaces, for example)

Suggested Integration Testing Methodology


System Integration Testing System Integration Testing

• We’ll talk about System Integration Testing later.

• For now, we should stick to the sequence of the Test lifecycle.

• Which means System Testing next.


System TestingSystem Testing

Context

Definition

Functional Systems testing

Non-Functional Systems Testing

Good Practices for System Testing



User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan



• System Testing - process of testing an integrated system to verify that it meets specified requirements

• Concerned with the behaviour of the whole system, not with the workings of individual components.

• Carried out by the Test Team

DefinitionDefinition


Functional System TestingFunctional System Testing

Definition

Requirements-based functional testing

Business process functional testing



A Functional requirement is (per IEEE):

‘A requirement that specifies a function that a system or system component must perform’

• A Requirement may exist as a text document and/or a model



Requirements-based functional testing - Functionality

Accuracy Provision of right or agreed results or effects

Interoperability Ability to interact with specified systems

ComplianceAdhere to applicable standards, conventions, regulations or laws

Auditability Ability to provide adequate and accurate audit data

SuitabilityPresence and appropriateness of functions for specified tasks



• testing against requirements and specifications

• test procedures and cases derived from:– detailed user requirements– system requirements functional specification– User documentation/instructions– high level System design

Requirements based testing



Requirements-based functional testing - techniques• Starts by using the most appropriate black-box testing techniques

• May support this with white-box techniques (e.g. menu structures, web page navigation)

• Risk based approach important



• test procedures and cases derived from:– expected user profiles– Business scenarios– use cases

• testing should reflect the business environment and processes in which the system will operate.

• therefore, test cases should be based on real business processes.

Business Process based testing



• “System testing is the most misunderstood and most

difficult testing process ”

Myers - 2004



Non-functional System TestingNon-functional System Testing

Definition

Non-functional requirements

Non-functional test types



• testing of those requirements that do not relate to functionality

Definition




• Emphasis on non-functional requirements:– Performance– Load– Data volumes– Storage– Recovery– Usability– Stress– Security*

• * Note that ISTQB treats this as a Functional test. From the syllabus:

– ‘Security Testing A type of functional testing, security testing, investigates the functions (e.g. a firewall) relating to detection of threats, such as viruses, from malicious outsiders.’




• The non-functional aspects of a system are all the attributes other than business functionality, and are as important as the functional aspects. These include:

– the look and feel and ease of use of the system

– how quickly the system performs

– how much the system can do for the user

• It is also about:– how easy and quick the system is to install

– how robust it is

– how quickly the system can recover from a crash


Non-functional test types

• Reliability - The capability of software to maintain its level of performance under stated conditions for a stated period of time. Is the software product reliable?

• Usability - Is the software product easy to use, learn and understand from the user’s perspective?

• Maintainability: The effort needed to make specified modifications. Is the software product easy to maintain?

• Efficiency: The relationship between the level of performance of the software and the amount of resources used, under stated conditions. Does the software product use the hardware, system software and other resources efficiently? Is the number of resources required by the software product during use affordable and acceptable?

• Portability: The ability of software to be transferred from one environment to another. Is the software product portable?

Non-functional System TestingNon-functional System TestingNon-functional System TestingNon-functional System Testing



Good Practices for System testing

• implement documented procedures for requirements analysis, control and traceability

• review deliverables to ensure feasible, testable requirements and associated acceptance criteria

• trace requirements to the design and tests which prove the requirement has been met

• test data, facilities and documentation must be sufficient to demonstrate conformance with requirements

• Test environment must closely mirror the target production system


System Integration Testing System Integration Testing

Context

Definition

Objectives

Interfaces to External Systems


System Integration TestingSystem Integration Testing

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan

System

Integration

Testing



• System Integration Testing is testing between the ‘System’ and ‘Acceptance’ phases.

• The System has already proven to be functionally correct, what remains to be tested is how the system reacts to other systems and/or organisations.

Definition



• The objective of System Integration Testing is to provide confidence that the system or application is able to interoperate successfully with other specified software systems and does not have an adverse affect on other systems that may also be present in the live environment, or vice versa

• It is possible that the testing tasks performed during System Integration Testing may be combined with System Testing, particularly if the system or application has little or no requirement to interoperate with other systems

• In terms of the V Model, Systems Integration Testing corresponds to the Functional and Technical Specification phases of the software development lifecycle

Objectives of Systems Integration Testing



• Having completed Component integration testing and Systems testing, one must execute the plan for system-to-system integration

• Infrastructure may need to be transformed in order to feed to an external system

• Black Box testing techniques used

Testing Interfaces to External Systems


Acceptance TestingAcceptance Testing

Context

Definition

User Acceptance Testing

Operational Acceptance Testing

Contract/Regulation acceptance testing

Alpha and Beta testing

Other Acceptance Test terms



User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan



• Acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

Definition



• Usually the responsibility of the Customer/End user, though other stakeholders may be involved. Customer may sub-contract the Acceptance test to a third party

• Goal is to establish confidence in the system/part-system or specific non-functional characteristics (e.g. performance)

• Usually for ensuring the system is ready for deployment into production

• May also occur at other stages, e.g.

– Acceptance testing of a COTS product before System Testing commences

– Acceptance testing a component’s usability during Component testing

– Acceptance testing a new significant functional enhancement/middleware release prior to deployment into System Test environment.

Definition



• Usually the final stage of validation

• conducted by or visible to the end user and customer

• testing is based on the defined user requirements

• Often uses the ‘Thread Testing’ approach:

• ‘A testing technique used to test the business functionality or business logic of the application in an end-to-end manner, in much the same way a User or an operator might interact with the system during its normal use.’

- Watkins 2001

This approach is also often used for Functional Systems Test - The same Threads serve both test activities

User Acceptance Testing (UAT)


• Often use a big bang approach

• black box testing techniques most commonly used

• Regression testing to ensure changes have not regressed other areas of the system

Acceptance TestingAcceptance TestingAcceptance TestingAcceptance Testing




• “If love is like an extended software Q.A. suite, then true love is like a final Acceptance Test – one often has to be willing to endure compromise, bug fixes and work-arounds;

otherwise, the software is never done ”

The Usenet Oracle




• The Acceptance of the system by those who have to administer it.

• Features covered include:– testing of backup/restore– disaster recovery– user management– maintenance tasks– periodic checks of security vulnerabilities

• ‘The objective of OAT is to confirm that the Application Under Test (AUT) meets its operational requirements, and to provide confidence that the system works correctly and is usable before it is formally "handed over" to the operation user. OAT is conducted by one or more Operations Representatives with the assistance of the Test Team’ 1 –Watkins 2001


Operational Acceptance Testing (OAT)


• Employs a Black Box Approach for some activities

• Also employs a Thread Testing approach – Operations representatives performing typical tasks that they would perform during their normal usage of the system

• Also addresses testing of System Documentation, such as Operations manuals


Operational Acceptance Testing (OAT)



• Contract Acceptance Testing - testing against the acceptance criteria defined in the contract

– final payment to the developer depends on contract acceptance testing being successfully completed

– acceptance criteria defined at contract time are often imprecise, poorly defined, incomplete and out-of-step with subsequent changes to the application

• Regulation Acceptance testing is performed against any regulations which must be adhered to, such as governmental, legal or safety regulations

Contract/Regulation Acceptance Testing



• early testing of stable product by customers/users

• feedback provided by alpha and beta testers

• alpha tests performed at developer’s site by customer

• beta tests conducted at the customer site by end user/customer

• published reviews of beta release test results can make or break a product (e.g. PC games)

Alpha & Beta Testing



• Factory Acceptance Testing (FAT)

• Site Acceptance Testing (SAT)

• Both address acceptance testing for systems that are tested before and after being moved to a customer’s site

Other Acceptance test terms


Test Types – The Targets of TestingTest Types – The Targets of Testing

Definitions

Functional Testing

Non-Functional Testing

Structural Testing

Confirmation & Regression Testing



• Target For Testing - A group of test activities aimed at verifying the software system (or a part of a system) based on a specific reason.

• Test type - A group of test activities aimed at testing a component or system regarding one or more interrelated quality attributes. A test type is focused on a specific test objective, e.g.

– reliability test

– usability test

– Structure or architecture of the system/software

– regression test

and may take place on one or more test levels or test phases.

Definitions



• A model of the software may be developed and/or used in structural and functional testing

• For example, in functional testing

– a process flow model

– a state transition model

– a plain language specification

• and for structural testing

– a control flow model

– a menu structure model

Definitions


Functional TestingFunctional Testing

• functional testing: Testing based on an analysis of the specification of the functionality of a component or system.

• ‘Specification’ – E.g. Requirements specification, Use Cases, Functional specification or maybe undocumented.

• ‘Function’ – what the system does

• Functional test – based on the Functions and features – may be applied at all Test levels (e.g. Component Test, System Test etc)

• Considers the external (not internal) behaviour of the software. Black- Box testing. What it does rather than how it does it. More on this later!


Non-Functional TestingNon-Functional Testing

• non-functional testing: Testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, interoperability, maintainability and portability

• May be performed at all Test levels (not just Non Functional Systems Testing)

• Measuring the characteristics of the system/software that can be quantified on a varying scale- e.g. performance test scaling


Structural TestingStructural Testing

• Structural testing: Testing based on an analysis of the internal structure of the component or system

• Also known as White Box Testing or Glass Box Testing

• May be performed at all Test levels but more commonly during Component Test and Component Integration Test

• Coverage measured as a % of items tested – i.e. how much the structure has been tested

• May be based on the system Architecture – e.g. a calling hierarchy

• Need for use of Test Tools – e.g. for testing coverage of Statements and Decision in the code

• More on White Box testing and Coverage later


Confirmation (Re-Testing) and Regression TestingConfirmation (Re-Testing) and Regression Testing

Re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions

Regression Testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed.


• Whenever a fault is detected and fixed then the software should be re-tested to show that the original fault has been fixed. This is known as Re-Testing.

• It is important that the test case is repeatable.

• In order to support this the test identifier should be included on the fault report.

• It is important that the environment and test data used are as close as possible as those used during the original test.

Confirmation Testing (Re-Testing)Confirmation Testing (Re-Testing)


• If the test is re-run and passes you cannot necessarily say the fault has been resolved because ..

• You also need to ensure that the modifications have not caused unintended side-effects elsewhere and that the modified system still meets its requirements – Regression Testing

• Regression testing should be carried out :-

– when the system is stable and the system or the environment changes

– when testing bug-fix releases as part of the maintenance phase

• It should be applied at all Test Levels

• It should be considered complete when agreed completion criteria for regression testing have been met

• Regression test suites evolve over time and given that they are run frequently are ideal candidates for automation

Regression TestingRegression Testing



• Selecting suitable tests involves :-– knowledge of the bug fixes and how they affect the system

– understanding the areas that have frequent faults

– understanding which areas of the system have undergone the most recent changes

– understanding the areas of the system which are most critical to the user

– understanding the core features of the system which must function correctly.

• The effectiveness of a regression test suite can diminish over time for a number of reasons :-

– tests are added for short term goals but not removed

– tests become redundant due to functionality changes

– test suite is not updated when major functionality changes are implemented

– execution time becomes prohibitively high

– maintenance of the test suite becomes prohibitively high.



• Reduction in effectiveness can be countered by :-

– maintaining cross references between system features and their corresponding tests

– monitoring the addition of tests to the suite

– Periodic review and removal of redundant tests

– review of the test suite when major enhancements are made to the system

– evaluation of the effectiveness of the test suite using metrics.



“ The probability of making an incorrect change is more than 50 %.

Much of this is due to overconfidence and ineffective or nonexistent software change testing.

We change just a couple of statements and believe we have not affected anything adversely.

We execute one case that tests the path that was changed and tell ourselves that the change has been tested.

IS IT, THEN, ANY WONDER THAT WE EXPERIENCE SO MANY PROBLEMS?!”

Hetzel 1998



Maintenance TestingMaintenance Testing

What is Maintenance testing?

Objectives of Maintenance testing

Problems of Maintenance testing

Concerns of Maintenance testing

How can we test changes?



• Maintenance testing: Testing the changes to an operational system or the impact of a changed environment to an operational system

• testing changes to a Live System

• Triggered by, for example,– Modification

• software upgrades

• Operating system changes

• system tuning

• emergency fixes

– software Retirement (may necessitate data archiving tests)

– Migration

• System migration (including operational tests of new environment plus changed software)

• database migration

What is Maintenance Testing?



• Develop tests to detect problems prior to placing the change into production

• Correct problems identified in the live environment

• Test the completeness of needed training material

• Involve users in the testing of software changes

Objectives of Maintenance Testing



• all that is available is the source code (usually with poor internal documentation and no record of testing) – poor or missing specifications

• program structure, global data structures, system interfaces and performance and/or design constraints are difficult to determine and frequently misinterpreted

• Baselined test plans and/or regression test packs often not updated

Problems of Maintenance testing



• will the testing process be planned?

• will testing results be recorded?

• will new faults be introduced into the system?

• will system problems be detected during testing?

• how much regression testing is feasible?

• will training be considered?

Concerns of Maintenance testing



How can we test changes?

• Maintenance testing involves testing what has been changed (i.e. Re-Testing)

• It also, importantly, utilises Impact Analysis as a method for determining what Regression testing is required for the whole system

• Traceability of Testware to source documents essential for effective impact analysis (we cover this more in a later topic)

• Scope of Maintenance tests based on Risk assessment – including size of change and size of system

• Maintenance testing may involve one or more test levels and one or more test types


Testing throughout the Software LifecycleTesting throughout the Software Lifecycle

• Firstly we looked at Software Development Models:– The V-Model – identifying the stages of testing, their relationship to the

Development stages and the type of work products involved

– Iterative Development Models, as used in RAD, RUP and Agile developments

– Also the characteristics that make for good testing in ANY life cycle model

– And that development models must be adapted to the context of project and product characteristics

• Next we talked about the different Test Levels:– Component Testing – testing of individual software components by the

development team

– Integration Testing at Component level – looking at different approaches such as Top-Down, Bottom-up and Big Bang

– System Testing

• Functional System Testing – requirements and business process based

• Non-Functional System Testing – testing the non-functional attributes of a system

– System (level) Integration Testing - testing that a system interoperates with other systems or organisations

SummarySummary


• And still under Test Levels ….– Acceptance Testing, comprising:

• User Acceptance Testing

• Operational Acceptance Testing

• Contract and Regulation testing

• Alpha and Beta testing

• Next we talked about Test Types– A group of testing activities aimed at testing one or more quality attributes

of the system, such as:

• Functional Testing

• Non-Functional Testing

• Structural testing

• Regression Testing

• Re-testing (confirmation)

– All Test Types can be performed at all test levels


SummarySummary


• An finally we talked about Maintenance testing

– Testing the changes (software and environmental) to an operational live system

– What the reasons (or triggers) are for Maintenance Testing

– We looked at the objectives of Maintenance Testing

– the problems that can be encountered during Maintenance testing

– and what we should consider for our Maintenance testing approach such as Impact Analysis and Regression testing


SummarySummary

• EDS Internal

03-23-05Version 2.3

Static TechniquesStatic Techniques



Static Testing – a Definition

Reviews and the Test Process

Review Process

Static Analysis

Summary



• Static testing techniques involve examination of the project’s documentation, software and other information about the software products without executing them

• Static Testing Includes both Reviews (e.g. of documentation) and Static Analysis of code

• Reviews, Static Analysis and dynamic testing have the same objective – identifying defects

• Static Testing and Dynamic Testing are complementary

• Each technique can find different types of defects effectively and efficiently

dynamic testing: Testing that involves the execution of the software of a component or system.

Static Testing – a DefinitionStatic Testing – a Definition


Reviews and the Test Process Reviews and the Test Process

Why Review

When and What to review

What do Reviews find

Benefits of reviews

TopicsTopics


Reviews and the Test ProcessReviews and the Test Process

• Find errors early in the development lifecycle (more cost effective to fix) More than 60% of the errors in a component can be detected using informal inspections (1)

• Reduce the errors in delivered systems

• Learning experience in standards and techniques

• Team building and motivation

• Improve the specification, design and code

• Education and training of developers and other project staff

• Determine the root causes of defects and measures for preventing recurrence

• Test and improve standards and procedures

• To assess the early stages of development for progress and completeness

Why ReviewWhy Review

1) Fagan, M. E., Advances in Software Inspections, IEEE Transactions on Software Engineering, 1987


0102030405060708090

100

Reqs Des Code Unit Accept Use

Cost


Why Review - The cost of errors Why Review - The cost of errors

Relative

Multiples


Requirements

Why Review -Why Review - Error/Fault PropagationError/Fault Propagation

correct

Requirement Errors

Design

correct

Design Errors

Code

correct

Coding Errors

(Faults)



Why Review -Why Review - Where errors are introducedWhere errors are introduced

0%

5%

10%

15%

20%

25%

30%

Reqs FunctDes

LogicalDes

Code Other

Errors



• “Quality control has as its mission the detection and elimination of defects in the product. Reviews are the front

line in this mission”

John W Horch 2003



Why Review Why Review



• A review could be done entirely as a manual activity

• Or there is tool support available

• The main manual activity is to examine a work product and make comments about it.

• Any software work product can be reviewed, including: – requirements and design specifications

– source code listings

– test plans, test cases, test scripts

– user documentation

– application administration and support material

– Web pages

• A software work product can be reviewed one or more times and using one or more review types

• Review everything as soon as possible

• Reviews start well before Dynamic Test execution

When and What to Review?When and What to Review?



• In contrast to dynamic testing, reviews find defects rather than failures

• Typical defects that are easier to find in reviews than in dynamic testing are:

– deviations from standards

– requirement defects

– design defects

– insufficient maintainability

– incorrect interface specifications.

What do Reviews Find?What do Reviews Find?



• Detect faults as they are introduced –i.e. early detection and correction• Reduce the risk of error propagation• Detect errors that Dynamic test execution unlikely to find, e.g.

requirement spec errors• Shorten development timescales• Reduce fault levels in delivered software• Lower cost and shorten testing timeframes• Lower cost over the life of the software• Create development productivity improvements• Reliably evaluates progress and capability (1)

• Educates and trains participants (1)

• Improve communication between project teams

(1) – The Complete Guide to Software Testing – Bill Hetzel

BenefitsBenefits


Review ProcessReview Process

Informal Reviews

Formal Review Types

Formal Review Process

Formal Review Roles and Responsibilities

Other Review Types

Key Success factors

TopicsTopics


• No Formal Review Process employed

• “Desk checking”, looking for possible problems

• The author of material checking his or her own material, possibly with one other peer (pair programming)

• Possibly a technical lead reviewing design and code

• Usually undocumented but useful, cheap and widely used

• This technique may be applied in low risk situations

• No metrics kept, review findings are not configured items

• Weaknesses – do not find as many faults as formal reviews

Review ProcessReview ProcessInformal ReviewInformal Review



• A walkthrough is a review of authored material led by the author and attended by a group of the author's peers (typically 2 to 6 peers). Primary purpose is education

• The material is presented by the author to the peer group, who focus on learning about the material, improving it and recording defects

• Peer group should include development, operation representatives, target audience, etc.

• Examples are Dry Runs or Scenario playing to validate product

• Sessions can be formal or informal

• Review sessions often open-ended (not time-boxed)

• Pre-meeting preparation often involved

• Weaknesses – do not find as many defects as technical reviews and inspections

Formal Review Types – WalkthroughsFormal Review Types – Walkthroughs


• May be performed as Peer Reviews without management participation

• Preferably lead by a trained moderator (not the author)

• Pre-Meeting preparation is required

• Primary purpose is to:

– Discuss

– make decisions

– evaluate alternatives

– find defects

– solve technical problems and check conformance to specifications and standards

• Degree of formality varies

• Reviewers bring a list of technical issues to the review

• Optional use of checklists and a review report

• During the meeting reviewers raise objections, ambiguities or inconsistencies in design or technical aspects being discussed

• Problems are clarified and documented - solutions are sought after the review has concluded

• Weaknesses – do not find as many faults as Inspections

Review ProcessReview ProcessFormal Review Types –Formal Review Types – Technical Technical ReviewsReviews


• Formal, systematic reviews of material. Primary purpose is fault finding and process improvement

• Led by an independent trained moderator (but not the author)

• Main purpose – find defects

• Attended by the author & the author's peers (usually 3 to 6) acting in defined roles

• Pre meeting preparation essential

• Follow a strict format– stated entry criteria and exit criteria

– seeking and recording defects

– use standardised rules, check-lists and techniques

– record metrics

• Formal follow-up process

• Optionally process improvement considerations part of review

• Weaknesses – expensive and time consuming but high defect yield!

Review ProcessReview ProcessFormal Review Types –Formal Review Types – InspectionsInspections


Planning

Kick-off

Review Overview – optional

Preparation

Review Meeting

Rework

Follow-up

Repeat Review - optional

Formal Review ProcessFormal Review ProcessFormal Review ProcessFormal Review Process



• Define Entry and Exit Criteria (for most formal reviews)

• Ensure that the volume of material to be reviewed is appropriate

• Identify roles, participants and establish a time and place for the review

PlanningPlanning



• Distribute the material to the participants

• Explain objectives, process and material to be reviewed

• Obtain copies of the relevant review and report templates

• Create checklists of areas to cover and distribute

– checklists can make reviews more effective and efficient

– E.g. a checklist based on perspectives such as user, maintainer, tester or operations

– or a checklist of typical requirements problems to focus on

• Make sure entry criteria has been/will be met

Kick OffKick Off


• Required for new or difficult material

• Overviews:– educate the participants

– allow participants to focus on technical content

– describe where the material fits in the system and in the development process

– focus on any complex functionality

– highlight any changes and explain the need for these changes


Review Overview Review Overview (Optional)(Optional)



• Each participant reviews the material to:

– learn about the material

– note suspected defects

– record questions

• In some circumstances, depending on the expertise of the participants, the moderator may ask certain participants to concentrate on particular aspects of the material during preparation

PreparationPreparation



• The material is read to the participants by the reader

• Defects are raised by the participants and recorded by the recorder

• Participants may make decisions about categorising and even handling the defects – though usually avoid ‘solutionising’

• Deliverables may include meeting minutes

• For Inspections - Pass or fail and repeat review decisions are usually made by the moderator

• The preparation time and the actual time may be recorded

Review MeetingReview Meeting


• The author must resolve all defects found during the review by reworking the material as recommended by the review report

• Note, the cost of rework is NOT included in the cost of reviews


ReworkRework


• Check the corrections to the material and account for all recorded defects

• If necessary, schedule a repeat review for the corrected material

• Inform management of the status of the corrected material

• Add the error data from the review to the project statistics database – enables process improvement!

• Complete and sign the review report and forms (Inspections)

• Ensure exit criteria met


Follow-upFollow-up


• If the material has been passed as is or if the rework is minor, no further reviews are required

• If a repeat review is required (e.g. if significant re-work was required) a repeat review must be scheduled with the same participants to verify the revised material


Repeat Review (optional)Repeat Review (optional)


Review ProcessReview ProcessFormal Review Roles and ResponsibilitiesFormal Review Roles and Responsibilities

Manager decides on the execution of reviews, allocates time in project schedules and determines if the review objectives have been met

Moderator the person who leads, plans and runs the review

May mediate between the various points of view and is often the person upon whom the success of the review rests

Author the writer or person with chief responsibility for the document(s) to be reviewed

Reviewers Individuals with a specific technical or business background (also called checkers or inspectors)

Identify and describe findings (e.g. defects)

Take part in any review meetings

Scribe/Recorder documents all the issues, problems and open points that were identified during the meeting


Review ProcessReview ProcessOther Review Types Other Review Types

• Some others not in the ISTQB Syllabus:

– Management reviews – reviews of plans, schedules, progress

– Audits – independent evaluation of conformance to standards, plans and procedures

– Post Implementation – review of project approach (including test approach)



– Each review has a clear predefined objective

– The right people are involved

– Defects found are welcomed (Authors must leave their ego at the door!)

– Defects are expressed objectively – no cornering the author – make it a positive experience!

– Review techniques are applied suitable to the review

– Checklists used if appropriate – focus the review effort

– Roles pre-defined – avoid duplication and increase effectiveness

– Training is given in review techniques- especially for Inspections

– Management buy in – schedule in review activities and effort

– There is an emphasis on learning and process improvement

Key Success FactorsKey Success Factors


Static AnalysisStatic Analysis

Definition

Description

Value

Types of Defects found

Use of Tools

TopicsTopics


Static AnalysisStatic Analysis

static analysis: Analysis of software artifacts, e.g. requirements or code, carried out without execution of these software artifacts.



• Objective - to find defects in software source code and software models

• Static analysis is performed without actually executing the software being examined by the tool whilst Dynamic testing does execute the software code

• Static analysis can locate defects that are hard to find in testing.

• As with reviews, static analysis finds defects rather than failures

• Static analysis tools analyze program code (e.g. using control flow graphing and data flow analysis techniques), as well as generated output such as HTML and XML.

Static Analysis Static Analysis Static Analysis Static Analysis

DescriptionDescription


• Early detection of defects prior to test execution

• Early warning about suspicious aspects of the code or design

• Calculation of metrics such as a high complexity measure

• Identification of defects not easily found by dynamic testing

• Detecting dependencies and inconsistencies in software models, such as links

• Improved maintainability of code and design

• Prevention of defects, if lessons are learned in development

• Cost effective – problems found earlier are cheaper to fix

• Static analysis is more effective and less expensive than dynamic testing (1)

• Static analysis is demonstrated to find 45% of expected errors before testing actually starts


ValueValue


Static Analysis Static Analysis

• Many complexity measures in common use, e.g.

– Cyclomatic Complexity – measures structural complexity

– Halstead Complexity – measures algorithmic complexity, measured by counting operators and operands

– Henry and Kafura metrics – measures coupling between modules (parameters, global variables, calls)

• Cyclomatic Complexity is the most widely used. It identifies how difficult the code is to understand, test and maintain. It also identifies the risk associated with testing a program

Value – Complexity AnalysisValue – Complexity Analysis



Types of Defects Types of Defects foundfound

• Unreachable code

• Undeclared variables

• Parameter type mismatches

• Uncalled functions and procedures

• Possible array bound violations

• Security Violations

• Inconsistent interface between modules and components

• Incorrect variable usage

• Syntax checking

• Violations of code standards

• Use of variables without first defining them

• variables that are declared but never used

• Use of variables after they have been “killed”



Use of ToolsUse of Tools

• Static analysis tools are typically used by developers • Used mainly before and during Component and Component Integration

testing• The tools check against predefined rules or programming standards• Also by designers during software modelling • Static analysis tools may produce a large number of warning messages• Hence the need to use the tools effectively (or can’t see the wood for

the trees!) • Compilers may offer some support for static analysis, including the

calculation of metrics


Static Techniques Static Techniques Static Techniques Static Techniques

SummarySummary

• We learned that Static Testing techniques involve examining documentation and software products without executing them

• That Static Testing covers both Reviews and Static Analysis

• For Reviews we learned about: – the different types of products that can be reviewed and that we review as

early as we can– The Benefits of conducting Reviews– And the typical defects that reviews uncover

• We learned about the Review process– The key characteristics of the 4 Review types, Informal, Walkthrough,

Technical and Inspection– The 6 main Stages that make up the Review process from Planning through

to Follow-up– The Roles and Responsibilities involved in that process– The Key Success Factors for Review – what makes them work


Static Techniques Static Techniques Static Techniques Static Techniques

SummarySummary

• And we learned about Static Analysis

– Analysis of software artefacts (e.g. code/design) without executing them– The Value of Static Analysis – what we gain by conducting it– The types of defects found in static analysis– And finally the use of tools such as a Compiler – who uses them and when

• EDS Internal

03-23-05Version 2.3

Test Design TechniquesTest Design Techniques


Test Design TechniquesTest Design TechniquesTest Design TechniquesTest Design Techniques

Identifying Test Conditions and Designing Test Cases

Categories of Test Design Techniques

Specification based or Black Box Techniques

Structure based or White Box Techniques

Experience Based Techniques

Choosing Test Techniques

Summary


Identifying Test Conditions and Designing Identifying Test Conditions and Designing Test CasesTest Cases

• Developing test material can be split into two distinct stages:– Defining “what” needs to be tested

– Defining “how” the system should be tested

• This process can vary from organisation to organisation, can be very formal or very informal with little documentation

• The more formal, the more repeatable the tests, but it does depend on the context of the testing being carried out

• The process of identifying test conditions and designing tests consists of the following steps:

– Defining test conditions

– Specifying test cases

– Specifying test procedures

– Developing a Test execution schedule


Test ConditionsTest Conditions

• Test Conditions are binary statements which determine a system’s fitness for purpose

• Defines what must be tested

• Sometimes referred to as a Test Item

• Grouped by Test Object or system function/process

• Test Requirement (Test basis ) documentation is analysed to determine the Test Conditions

• Test Conditions are then cross referenced to one or more test cases for execution

• Not all Test Conditions are as important as others so each Test Condition is assigned a risk

• Test Conditions should be linked back to their source documents from which they are derived. This helps for two reasons:

– Impact Analysis

– Traceability



• A Test Case defines how the system should be tested

• They typically contain

– Input values

– Execution pre conditions

– Expected results (output, changes in state etc)

– Post conditions

– Cross referenced test conditions

• Remember expected results must be defined before execution

• There can be many Test Cases developed to test a single Test Condition

Test CasesTest Cases



Expected ResultsExpected Results

• “A necessary part of a test case is a definition of the expected output or result”


Myers - 1979


• The Test Procedures Specification specifies the sequence of actions for a test, i.e. one or more Test Cases

• It is also known as a Test Script

• The Test Script can be manual or automated

• Contents of a Test Procedure are:

– Test procedure specification identifier

– Purpose

– Special requirements

– Procedure steps

Test ProceduresTest Procedures



• The Test Procedure Specifications (i.e. Test Scripts) are subsequently included in a Test Execution Schedule

• This schedule defines the order in which the test scripts are executed, when they are to be carried out and by whom

• The Execution schedule will also need to take account of:

– Regression Tests

– Prioritisation

– And technical and logical dependencies

Test Execution ScheduleTest Execution Schedule



Test Execution Schedule

Test Conditions, Cases, Procedures and ScheduleTest Conditions, Cases, Procedures and Schedule

How

Test CasesTest

Cases

Source

d D

ocu

menta

tion

What

Test Condition

Priority

WhenHow

Test Procedure

Specification

Manual Test Script

Automated Test Script

or

Test Procedure

Specifications


Categories of Test Design TechniquesCategories of Test Design TechniquesCategories of Test Design TechniquesCategories of Test Design Techniques

Definition of Black Box Testing

What is black box testing?

Definition of White Box Testing

What is white box testing?

Black, White and Experience based (comparison)

Use of Techniques and tools


Black and White Box TestingBlack and White Box Testing

Definition of Black Box Testing

• Black-box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system.


What is Black Box Testing?

– testing without knowing the internal workings of the code

– WHAT a system does, rather than HOW it does it

– typically used at System Test phase, although can be useful throughout the test lifecycle

– also known as specification based testing

– applies for Functional and Non-Functional testing

Black and White Box TestingBlack and White Box TestingBlack and White Box TestingBlack and White Box Testing

Input Output

If Output = Expected result then pass



Definition of White Box Testing

white-box testing: Testing based on an analysis of the internal structure of the component or system.



What is White Box Testing?

– testing based upon the structure of the code

– typically undertaken at Component and Component Integration Test phases by development teams

– also known as structural or glass box testing or structure based testing

Input Output


Black, White and Experienced basedBlack, White and Experienced based

Black(Specification Based)

•Based on requirements•From the requirements, tests are created•Specification Models can be used for systematic test case design

Techniques•Equivalence Partitioning•Boundary Value Analysis•Decision Tables•State Transition Testing•Use Case Testing

White(StructureBased)

•Based on code and the design of the system•The tests provide the ability to derive the extent of coverage of the whole application

Techniques•Statement coverage•Branch Coverage•Decision Coverage

Experience(Black box)

Techniques•Error Guessing•Exploratory Testing

•Based on the knowledge of the tester•Using past experienced use & intuition to “guess” where errors may occur


Use of Techniques and Tools

– techniques provide a systematic method of approaching software testing

– enable tests to be repeatable

– provide a measure of software coverage

– due to the scale and complexity of systems, tools are often used to assist with testing especially white box testing

Black and White Box TestingBlack and White Box TestingBlack and White Box TestingBlack and White Box Testing


Specification based or Black Box TechniquesSpecification based or Black Box TechniquesSpecification based or Black Box TechniquesSpecification based or Black Box Techniques

Equivalence Partitioning

Boundary Value Analysis

Decision Table Testing

State Transition Testing

Use Case Testing

Other black box test techniques



• aim is to treat groups of inputs as equivalent and to select one representative input to test them all

• Best shown in the following example….

– If we wanted to test the following IF statement:

– ‘IF VALUE is between 1 and 100 (inclusive) (e.g. VALUE >=1 and VALUE <= 100) THEN …..’

– We could put a range of numbers as shown in the next slide through test cases

Black Box Test TechniquesBlack Box Test TechniquesBlack Box Test TechniquesBlack Box Test Techniques



1 10099

87

65

48

37

19 53

IF Value >= 1 AND Value <= 100 THEN ….

IN RANGE

0

-1

101

OUT OF RANGE

OUT OF RANGE

1000



• the numbers fall into a partition where each would have the same, or equivalent, result i.e. an Equivalence Partition (EP) or Equivalence Class

• EP says that by testing just one value we have tested the partition (typically a mid-point value is used). It assumes that:

1) if one value finds a bug, the others probably will too2) if one doesn't find a bug, the others probably won't either




• in EP we must identify Valid Equivalence partitions and Invalid Equivalence partitions where applicable (typically in range tests)

• the Valid partition is bounded by the values 1 and 100

• plus there are 2 Invalid partitions




101

1000

1 10099

87

65

48

37

19 53

IF Value >= 1 AND Value <= 100 THEN ….

0

-1

‘VALID’ PARTITION

‘INVALID’ PARTITION

‘INVALID’ PARTITION




• Time would be wasted by specifying test cases that covered a range of values within each of the three partitions, unless the code was designed in an unusual way

• There are more effective techniques that can be used to find bugs in such circumstances (such as code inspection)

• EP can help reduce the number of tests from a list of all possible inputs to a minimum set that would still test each partition




• If the tester chooses the right partitions, the testing will be accurate and efficient

• If the tester mistakenly thinks of two partitions as equivalent and they are not, a test situation will be missed

• Or on the other hand, if the tester thinks two objects are different and they are not, the tests will be redundant

• EP can be used for all Levels of Testing

• EP is used to achieve good input and output coverage, knowing exhaustive testing is often impossible

• It can be applied to human input, input via interfaces to a system, or interface parameters in integration testing





• Boundary Value Analysis (BVA) uses the same analysis of partitions as EP and is usually used in conjunction with EP in test case design

• As with EP, it can be used for all Test levels

• BVA operates on the basis that experience shows us that errors are most likely to exist at the boundaries between partitions and in doing so incorporates a degree of negative testing into the test design

• BVA Test cases are designed to exercise the software on and at either side

of boundary values



For our previous example of ‘Value >= 1and Value <= 100’

Value = 0 (invalid)

Value = 1 (valid)

Value = 2 (valid)

Value = 99 (valid)

Value = 100 (valid)

Value = 101 (invalid)




• find the boundary and then test one value above and below it

• ALWAYS results in two test cases per boundary for valid inputs and three tests cases per boundary for all inputs

• inputs should be in the smallest significant values for the boundary (e.g. Boundary of ‘a > 10.0’ should result in test values of 10.0, 10.1 & 10.2)

• only applicable for numeric (and date) fields




Decision Table Decision Table TestingTesting

• Table based technique where

– Inputs to the system are recorded

– Outputs to the system are defined

• Inputs are usually defined in terms of actions which are Boolean (true or false)

• Outputs are recorded against each unique combination of inputs

• Using the Decision Table the relationships between the inputs and the possible outputs are mapped together

• As with State Transition testing, an excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels

• Especially useful for complex business rules



Decision Table StructureDecision Table Structure

Test 1 Test 2 Test 3

Input 1 T T F

Input 2 T T F

Input 3 T DON’T CARE F

Input 4 T F T

Response 1 Y Y N

Response 2 Y N Y

Response 3 N Y N

Inputs / Actions

Output / Response

Each column of the table corresponds to a business rule that defines a unique combination of conditions that result in the execution of the actions associated with that rule

The strength of Decision Table testing is that it creates combinations of conditions that might not otherwise have been exercised during testing



Decision Table ExampleDecision Table Example

Test 1 Test 2 Test 3

> 55 yrs old F T T

Smoker F T F

Exercises 3 times a week + T F T

History of Heart Attacks F T F

Insure Y N Y

Offer 10% Discount N N Y

Offer 30% Discount Y N N

What will be the out come of the following Scenarios?

Joe is a 22 year old non smoker who goes to the gym 4 times / week and has no history of heart attacks in his family

Kevin is 62 year old non smoker who swims twice a week and plays tennis. He has no history of heart attacks in his family



State Transition State Transition TestingTesting

• State Transition Testing uses the following terms:– state diagram: A diagram that depicts the states that a component or system

can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610]

– state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions.

– state transition: A transition between two states of a component or system.

– state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. Also known as N-switch testing.

• An excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels

• Often used in testing:– Screen dialogues

– Web site transitions



State Transition State Transition DiagramDiagram

State A

State B

Event/Action etc

Starting State

End State

Transition Between start and end states

Event from outside the systemAction triggered by Event



State Transition Example Simplified Car State Transition Example Simplified Car GearsGears

ReverseNeutral

1st Gear

2nd Gear

3rd Gear

Change Up/AccelerateChange Up/

Accelerate

Change Up/Accelerate

Change Down/Decelerate

Change Down/Move Back


Change Up/Accelerate




State Transition - State Transition - Switch CoverageSwitch Coverage

• Switch Coverage is a method of determining the number tests based on the number of “hops” between transitions. Some times known as Chow

0-Switch Coverage (1 hop)RNNRN11N12212332

1-Switch Coverage (2 hops)RN1RNRNRNN12N1N1NR Etc.

These hops can be used to determine the VALID tests



State Transition - State State Transition - State TableTable

• While Switch testing helps determine the valid tests, we also need to look for the invalid tests.

Change Up Change Down

R Acc / Neutral Null

N Acc/1st Gear Dec / Reverse

1st Acc/ 2nd Gear Dec / Neutral

2nd Acc / 3rd gear Dec / 1st Gear

3rd Null Dec / 2nd Gear

Invalid tests are those identified by a null output in this case

• Changing down from reverse• Changing up from 3rd



State Transition – Another State Transition – Another example for a Theatre Show example for a Theatre Show reservationreservation


Show Options provided

Request Show Options

Show selected

Choose Show

Change Mind/

Return to Options

Show Reservation

Made

Reserve Show

Show Reservation

Paid For

Pay for Show

Ticket Received

Issue

Ticket

Cancelled Reservation

Cancel reservation

Cancel reservation (return ticket)/Issue Refund

Cancel reservation/ Issue Refund


State Transition – Another type of State TableState Transition – Another type of State Table


Current State

Event/Next State

Event/ Next State

Event/ Next State

Event/ Next State

Event/ Next State

Event/ Next State

A B C D E F

SS 1

1 2

2 1 3

3 4

4 ES

ES


Use Case TestingUse Case Testing

• Use cases are a method of describing requirements

• Structured approach to requirements design

• Usually has one main flow (though may also have alternate flows)

• Each use case is a process flow or scenario

• Particularly useful in determining tests for (Functional) Systems Test and for UAT

• Also good at detecting defects in the integration of interfaces

• The main parts of a Use Case are:– Actors users of the system

– Pre conditions What are the starting requirements for the use case

– Post conditions The state the system will end up in once completed



• Syntax testing

— test cases are prepared to exercise the rule governing the format of data in a system (e.g. a Zip or Postal Code, a telephone number)

• Random testing

— test cases are selected, possibly using a pseudo-random generation algorithm, to match an operational profile

Other Black Box Test TechniquesOther Black Box Test TechniquesOther Black Box Test TechniquesOther Black Box Test Techniques


Structure Based or White Box Test TechniquesStructure Based or White Box Test TechniquesStructure Based or White Box Test TechniquesStructure Based or White Box Test Techniques

Statement Testing

Decision Testing

Assessing Completeness (Coverage)

Other White Box test techniques


– aim is to display that all executable statements have been run at least once

– coverage measurement is:

number of statements executed

total number of statements

White Box Test TechniquesWhite Box Test TechniquesWhite Box Test TechniquesWhite Box Test Techniques

Statement TestingStatement Testing


1. Read vehicle

2. Read colour

3. If vehicle = ‘Car’ Then

4. If colour = ‘Red’ Then

5. Print “Fast”

6. End If

7. End If

1

2

7

3

4

5

6

Example 1


Statement TestingStatement TestingControl Flow Graph


1. Read A

2. If A > 40 Then

3. A = A * 2

4. End If

5. If A > 100 Then

6. A = A – 10

7. End If

1

3

7

6

4

2

5

Example 2




1. Read bread

2. Read filling

3. If bread = ‘Roll’ Then

4. If filling = ‘Tuna’ Then

5. Price = 1.50

6. Else

7. Price = 1.00

8. End If

9. Else

10. Price = 0.75

11. End If

1

2

4/6

57

11

8

3/9

Example 3



10


– Aim is to demonstrate that all Decisions have been run at least once– Coverage measurement is:

the number of Decisions executed

the total number of Decisions

– Note that Decision testing is related to Branch testing – Decisions are lines of code where there are two or more alternative routes based

on a decision that has to be made, i.e. IF/THEN/ELSE– Branches are where there are one of two or more alternative routes from a line of

code but also include JUMPs and GO TOs – Decision and Branch Test coverage for a piece of code is often the same, but not

always – We are only interested in Decision coverage though. – Note BT is always >=ST


Decision TestingDecision Testing


1. Read vehicle

2. Read colour

3. If vehicle = ‘Car’ Then

4. If colour = ‘Red’ Then

5. Print “Fast”

6. End If

7. End If

1

2

7

3

4

5

6

Example 1




1

3

7

6

4

2

5

Example 2


1. Read A

2. If A > 40 Then

3. A = A * 2

4. End If

5. If A > 100 Then

6. A = A – 10

7. End IfC



1. Read bread

2. Read filling



5. Price = 1.50

6. Else

7. Price = 1.00

8. End If

9. Else

10. Price = 0.75

11. End If

1

2

4/6

57

10

11

8

3/9

Example 3





Statement and Decision TestingStatement and Decision Testing

The pseudo-code can be expressed in Flow-chart format

1. Read A

2. If A > 40 Then

3. A = A * 2

4. End If

5. If A > 100 Then

6. A = A – 10

7. End If


Assessing Completeness (Coverage)Assessing Completeness (Coverage)1. Read bread

2. Read filling



5. Price = 1.50

6. Else

7. Price = 1.00

8. End If

9. Else

10. Price = 0.75

11. End If

We already know:

Statement Coverage = 3

Decision Coverage = 3

Number of executable Statements = 7

Number of Branches = 4 (T and F of each IF)



Assessing Completeness (Coverage)Assessing Completeness (Coverage)

• Based on the following test set:

• What is the test Statement Coverage and Decision Coverage?

1. Read bread

2. Read filling



5. Price = 1.50

6. Else

7. Price = 1.00

8. End If

9. Else

10. Price = 0.75

11. End If

Bread Filling

Roll Tuna

Sandwich Ham



Other White Box Test TechniquesOther White Box Test Techniques

• Branch Condition testing— Branch Condition Testing requires that the True and False of each Boolean operand

is tested (Boolean Operands in this example: If A > 30 and B >= 5)

• Branch Condition Combination testing— Branch Condition Combination Coverage would require all combinations of Boolean

operands to be evaluated

• Modified Condition Decision testing— Modified Condition Decision Coverage requires test cases to show that each

Boolean operand can independently affect the outcome of the decision

• Dataflow testing— Data flow testing aims to execute sub-paths from points where each variable in a

component is defined to points where it is referenced.

• Linear Code Sequence And Jump (LCSAJ) testing— LCSAJ testing requires a model of the source code which identifies control flow

jumps (where control flow does not pass to a sequential statement).



Experience Based TechniquesExperience Based TechniquesExperience Based TechniquesExperience Based Techniques

Definition

Error Guessing

Exploratory Testing


Experienced Based TechniquesExperienced Based Techniques

• Uses the knowledge of people and the experience of past projects to determine likely errors based on the knowledge

– Testers

– Users

– Stakeholders

• Experience based techniques are non structured and do not rely on specification documents. This makes them unable to be measured in terms of coverage



Experience Based TechniquesExperience Based Techniques

• Tests are derived from skill and intuition

• Used to AUGMENT more systematic methods

• Good to identify tests which may not be explicitly described in the specifications

• Most beneficial when applied after more formal measures

• The success of these techniques is directly related to the knowledge and skill of the testers

• Does not provide any form of measurement or coverage



• Using experience to postulate errors

• Use Error Guessing to complement test design techniques

• Use as a “mopping up” approach to supplement systematic techniques

• Can be useful to identify special tests not easily captured by formal techniques, especially when applied after more formal approaches

• So don’t use as a first choice technique!

• Structured approach to error guessing

• Create a list of all possible errors

• Then create tests to attack these errors

• Remember these defect attack lists are built on experience, previous defects and from common knowledge as to why systems fail

Error GuessingError GuessingError GuessingError Guessing



• Error Guessing tests may include

• ‘Enter 00000 or 99999 in to a field’

• Creating surnames with quotes in, such as O’Donnell

• Nulls in mandatory fields

• Reserved characters ($%& for web systems)

Error GuessingError GuessingError GuessingError Guessing



Exploratory testingExploratory testing• Exploratory testing is a concurrent process where

– Test design, execution and logging happen simultaneously

– Testing is often not recorded

– Makes use of experience, heuristics and test patterns

– Testing is based on a test charter that may include

• Scope of the testing (in and out)

• Why? - what is the focus of your testing

• A brief description of how tests will be performed

• Expected problems

– Is carried out in time boxed intervals

• More structured than Error guessing

Risk Analysis CharterExploratory

Sessions

Notes

Debriefing



Choosing test TechniquesChoosing test Techniques

• How do you chose the right technique?– Type of system

– Standards

– Customer or contractual requirements

– Level of risk

– Type of risk

– Testing objectives

– Documentation available

– Knowledge / skills of the testers

– Time and budget

– Development processes

• Pick the right techniques for the right situation


Test Design Techniques - SummaryTest Design Techniques - Summary

• We learned about

– Identifying Test Conditions

– Designing Test Cases from the Test Conditions

– Creating Test Procedure Specifications to sequence our Test Cases

– Creating Test Execution schedules to define the order in which the test scripts are executed, when they are to be carried out and by whom

– The importance of traceability to requirements and specification of expected results

• We learned about the difference between Black and White Box testing

– White Box (Structure-based) Testing is based upon the structure of the program code

– Black Box (Specification Based) Testing is without reference the internal working of the program code

– The reasons why both are useful


Test Design Techniques - SummaryTest Design Techniques - Summary

• We learned about the different Black box techniques, mainly:

– Equivalence Partitioning

– Boundary Value Analysis

– State Transition Testing

– Decision Tables

– Use Case Testing

– Experience based Testing

• We learned about the different White box techniques, mainly:

– Statement Testing– Decision Testing

• For all Black and White box techniques we learned why they are of use and for which test levels they are typically applied

• EDS Internal

03-23-05Version 2.3

Test ManagementTest Management


Test ManagementTest Management

Test Organisation

Test Planning and Estimation

Test Progress Monitoring and Control

Configuration Management

Risk and Testing

Incident Management

Summary


Test OrganisationTest Organisation

Test Independence

Testing Roles Within the Team

The Test Leader

The Tester


Test IndependenceTest Independence

• More effective for someone other than the developer to test the system

– More impartial

– No preconceived ideas about what requires testing

– No bias or emotional attachment

Levels of Test Independence:

1. Independent testers within the development teams.

2. Independent test team or group within the organization, reporting to project management or executive management.

3. Independent testers from the business organization, user community and IT.

4. Independent test specialists for specific test targets such as usability testers, security testers or certification testers

5. Independent testers outsourced or external to the organization.

Levels of Independence


Test IndependenceTest Independence• Most test projects require multiple levels of testing where some of the

testing is carried out by independent testers.

Acceptance Testing

System Integration

Testing

System Testing

Component Integration Testing

Component Testing

Developers

Independent Testers


Benefits

• Independent testers see other and different defects, and are unbiased

• An independent tester can verify assumptions people made during specification and implementation of the system

• Usually a Cost saving – Better skills = more effective testing and fewer defects getting into production

– For Third Party test outsourcing, ‘better to rent than to own’

Drawbacks

• Isolation from the development team (if treated as totally independent).

• Independent testers may be the bottleneck as the last checkpoint

• Developers lose a sense of responsibility for quality

• Can be a greater cost – need to consider viability

• For Third Party test outsourcing, the project carries the risk

Independent testing benefits & drawbacks

Test IndependenceTest Independence


Testing Roles within the TeamTesting Roles within the Team

• Where possible some or all of the testing should be carried out by independent testers

• Developers may contribute at the lower levels (typically Component Testing and maybe Component integration)

• Specialist testers mainly play a role at Functional, Non Functional and System integration test levels

• The Business and System Administrators provide testing for Acceptance testing (e.g. UAT, OAT)

• The Test processes and rules are often defined by the Independent testers but must be agreed by management

Independent testing – who does what


The Test LeaderThe Test Leader

• Also known as Test Manager or Test Coordinator

• This role may also be done by

– Project manager

– QA manager

– Development manager

• In large projects this role may be split into a

– Test Manager

– Test (team) Leader

• Key tasks of a leader

– Test Planning

– Test Monitoring

– Test Control

Role of the Test Leader


The Test Leader The Test Leader

• Strategy & Management– Write and review the test strategy

– Coordinate the test strategy

– Plan testing effort – context, risks & approach

– Proactive representation in other project activities – ensure testing has correct focus

– Ensure proper configuration management of Testware exists

– Determine what should be automated

– Select and implement the most appropriate tools for testing including necessary training

– Management and definition of the test environmental requirements

– Define the test schedule based on the delivery of code in to test

• Monitor– Define, record and continually review the testing project metrics

– Monitor test progress against the test schedule

– Write the test summary reports

• Control– Adapt testing effort based results and progress

The Test Leader’s tasks


The Tester The Tester

• Make up the majority of the resources in the testing group.

• May be specialists in a particular area – Automation

– Performance

– Usability

– Security

• Or alternatively may work more generally doing:– Test Analysis

– Test Design

– Test Execution

– Test Environment management

– Test Data management

• Typically testers at the component level are developers

• At the Acceptance level testers are typically business experts or users or operators (for operational acceptance testing)

The role of the tester


The Tester The Tester

• Review and contribute to test plans.

• Analyse, review and assess user requirements, specifications and models for testability.

• Create test specifications.

• Set up the test environment (often coordinating with system administration and network management).

• Prepare and acquire test data.

• Implement tests on all test levels, execute and log the tests, evaluate the results and document the deviations from expected results.

• Use test administration or management tools and test monitoring tools as required.

• Automate tests (may be supported by a developer or a test automation expert).

• Measure performance of components and systems (if applicable).

• Review tests developed by others.

The tester’s tasks


Test Planning and EstimationTest Planning and Estimation

Test Planning

Test Planning Activities

Exit Criteria

Test Estimation

Test Approaches


Test PlanningTest Planning

• All projects require a set of plans and strategies which define how the testing will be conducted.

• There are number of levels at which these are defined:

Test Policy

Master TestPlan/Test Strategy

FunctionalTest Plan

UAT Test Plan

System IntegTest Plan

Defines how the organisation will conduct testing

Defines how the project will conduct testing

Defines how each level of testing will be conducted

Test strategies and test plans



• Factors which affect test planning– The organisation’s test policy

– Scope of the testing being performed

– Testing objectives

– Project Risks – e.g. business, technical, people

– Constraints – e.g. business imposed, financial, contractual etc

– Criticality (e.g. system/component level)

– Testability

– Availability of resources

• Test plans are continuously refined– As more information becomes available

– As new risks arise or others are mitigated

– Not set in concrete, but changes must be carefully managed

Test planning factors


Test Plan Contents (IEEE 829)Test Plan Contents (IEEE 829)1. Test Plan Identifier

2. Introduction

3. Test Items

4. Features to be Tested

5. Features not to be Tested

6. Approach

7. Item Pass/Fail Criteria

8. Suspension Criteria and Resumption Requirements

9. Test Deliverables

10. Testing Tasks

11. Environmental Needs

12. Staffing and Training Needs

13. Responsibilities

14. Schedule

15. Planning Risks and Contingencies

16. Approvals



Test Planning ActivitiesTest Planning Activities

• Approach - Defining the overall approach of testing (the test strategy), including the definition of the test levels and entry and exit criteria.

• Integrating and coordinating the testing activities into the software life cycle activities: acquisition, supply, development, operation and maintenance.

• Making decisions about: – what to test – who – i.e. what roles will perform the test activities– when and how the test activities should be done and when they should be stopped (exit

criteria – see next slides) – how the test results will be evaluated

• Assigning resources for the different tasks defined.• Testware definition- Defining the amount, level of detail, structure and templates

for the test documentation.• Selecting metrics for monitoring and controlling test preparation and execution,

defect resolution and risk issues.• Process - Setting the level of detail for test procedures in order to provide enough

information to support reproducible test preparation and execution.

Defining a test plan


Exit CriteriaExit Criteria

How do we know when to stop testing?

When our exit criteria have been met?

Run out of budget?

Boss says stop?

All defects have been fixed?

Run out of time?

The business tells you it went live last night!


Exit CriteriaExit Criteria

• Purpose of exit criteria is to define when we STOP testing either at the:

– End of all testing – i.e. product Go Live

– End of phase of testing (e.g. hand over from System Test to UAT)

• Exit Criteria typically measures:

– Thoroughness measures, such as coverage of requirements or of code or risk coverage

– Estimates of defect density or reliability measures. (e.g. how many defects open by category)

– Cost.

– Residual Risks, such as defects not fixed or lack of test coverage in certain areas.

– Schedules - such as those based on time to market.

What are exit criteria?


Test Estimation Test Estimation

• estimates provide predictions of effort and cost for conduct of testing activities that support :

– establishment of budgets

– procurement of resources

– production of a schedule for use in tracking and control

• by definition estimates are not exact and good estimates are ones that provide predictions within realistic tolerance limits

Test Estimation - Why


Test EstimationTest Estimation

• Estimation based on metrics from similar or previous projects or typical values

• Use metrics such as

– Length of time in preparation

– Length of time in execution

– Defect turn around time

– Down time and delays

• However it is important that you compare metrics from similar sized projects.

• Size of project is also required

– Function point analysis

– LoC (lines of code)

Test Estimation - How



• Estimating tasks by the owner of these tasks or by experts

• Experts may use formal metrics

• Or may be more gut feel and experience based

Test Estimation - How


Estimation - FactorsEstimation - Factors

• The testing effort can depend on:

– Characteristics of the product:

• the quality of the specification

• the size of the product

• the complexity of the problem domain

– Characteristics of the development process:

• the stability of the organization

• tools used

• test process

• skills of the people involved

• time pressure

– The outcome of testing

• the number of defects

• the amount of rework required.

• Once estimation is complete the resource requirements and projects schedules can be finalised




• “Test estimation is hard to do perfectly, but not terribly hard to do well ”

Black - 2002



Test ApproachesTest Approaches

• One method of classifying the way testing is done is by looking at when the bulk of testing is carried out

Preventative Reactive

Code Delivered Go LiveRequirements Delivered

Test Designdone here

Test Designdone here



• Analytical approaches, such as risk-based testing where testing is directed to areas of greatest risk.

• Model-based approaches, such as stochastic testing using statistical information about failure rates (such as reliability growth models) or usage (such as operational profiles).

• Methodical approaches, such as failure based (including error guessing and fault-attacks), check-list based, and quality characteristic based.

• Process- or standard-compliant approaches, such as those specified by industry-specific standards or the various agile methodologies.

• Dynamic and heuristic approaches, such as exploratory testing where testing is more reactive to events than pre-planned, and where execution and evaluation are concurrent tasks.

• Consultative approaches, such as those where test coverage is driven primarily by the advice and guidance of technology and/or business domain experts outside the test team.

• Regression-averse approaches, such as those that include reuse of existing test material, extensive automation of functional regression tests and standard test suites.

More ApproachesMore Approaches


Approaches CombinedApproaches Combined

• A preventative, Risk & Process based approach – would start testing early and use a standard industry approach such as the V model using the defined risks as a basis for testing

or

• A reactive, heuristic approach - would start after the code has been delivered and use unplanned testing techniques such as exploratory testing



Selection of Approaches - Selection of Approaches - Choosing the right approach

• The selection of a test approach should consider the context, including:

– Risk of failure of the project, hazards to the product and risks of product failure to humans, the environment and the company

– Skills and experience of the people in the proposed techniques, tools and methods

– The objective of the testing endeavour and the mission of the testing team

– Regulatory aspects, such as external and internal regulations for the development process

– The nature of the product and the business



Test Progress Monitoring and ControlTest Progress Monitoring and Control

Test Progress Monitoring

Test Reporting

Reporting and Interpreting Test metrics

Test Control


Test Progress MonitoringTest Progress Monitoring

• Need to know the status of the testing project at any given point in time

• Need to provide visibility on the status of testing to other stake holders

• Need to be able to measure your testing against your defined exit criteria

• Need to be able to assess progress against

– Planned schedule

– Measure how you are tracking against your defined budget

Test Monitoring - Why


Test Progress Monitoring Test Progress Monitoring

• Typical Metrics include

– Test preparation - Percentage of work done in test case preparation (or percentage of planned test cases prepared).

– Test environment preparation - Percentage of work done in test environment preparation.

– Test case execution (e.g. number of test cases run/not run, and test cases passed/failed).

– Defect statistics (e.g. defect density, defects found and fixed, failure rate, and retest results).

– Test coverage -e.g. % of requirements tested, risks or code coverage.

– Dates of test milestones – monitoring the test schedule

– Testing costs, including the cost compared to the benefit of finding the next defect or to run the next test.

– Subjective confidence of testers in the product.

Test Monitoring - How


Test ReportingTest Reporting

• Summarizes the testing carried out

– What happened during testing, for example:

• Were the dates met?

• Was the Test Plan adhered to and if not, what changes were made and why

• What problems were encountered and how were these addressed

– Analysed metrics from testing, for example:

• Defects remaining unresolved along with associated risks and resolution plans

• Number of tests run, not run, passed, failed etc

Test Reporting - What


Test ReportingTest Reporting

• Typical Contents are (IEEE 829)

– Test summary report identifier;

– Summary

– Variances;

– Comprehensive assessment;

– Summary of results;

– Evaluation;

– Summary of activities;

– Approvals.

Test Reporting - What


Reporting and Interpreting MetricsReporting and Interpreting Metrics

• Metrics should be collected during and at the end of a test level in order to assess:

– The adequacy of the test objectives for that test level.

– The adequacy of the test approaches taken.

– The effectiveness of the testing with respect to its objectives.

– Metrics are a valuable input into process improvement

Test Reporting - How


Interpreting Metrics – testing progressInterpreting Metrics – testing progress• On the face of it things look good

• Can we go live?

• What else would we need to know?

• However…

• What are the risks of the parts that have failed

• Does this chart account for all the testing scheduled – is there more to come?

Pass

Not Tested

Failed



Interpreting Metrics – Defect reportingInterpreting Metrics – Defect reporting

0

20

40

60

80

100

120

140

160

d1 d2 d3 d4 d5 d6

found

fixed

Time

No. D

efe

cts

Gap between = number of Incidents still outstanding




“…Testing is a measurement of software quality.”

Bill Hetzel 1988



Test ControlTest Control

• Test control is the response to Test Monitoring and Test Reporting that allows us to be IN CONTROL of the project

• Projects do go awry and issues occur. We need Monitoring and Reporting to know whether they are likely to go, or have already gone, awry.

• The process of Control is the corrective actions required to put a testing effort (project) back on track.

• For Example;

– Re-prioritize tests when an identified risk occurs (e.g. software delivered late).

– Change the test schedule due to availability of a test environment.

– Set an entry criterion requiring fixes to have been retested by a developer before accepting them into a build.

Test Control - Why


Configuration ManagementConfiguration ManagementConfiguration ManagementConfiguration Management

Symptoms of Poor CM

Configuration Items and their control


• Can’t match source and object code

• Can’t identify source code changes made for a particular version of software

• Simultaneous changes to source code by more than one programmer (changes lost)

• Old bugs previously fixed reappear

• Incorrect (e.g. out of date) versions of documents or other objects are used

• You have no control over the composition of the test environment (hardware and software items)

Symptoms of poor CM


• The purpose of Configuration Management is to establish and maintain the integrity of the products (components, data and documentation) of the software or system through the project and product life cycle.

• Items such as:

– plans, specifications (requirements, design)

– data dictionaries and cross-references

– software modules (purchased or developed)

– user and maintenance documentation

– support software (libraries, databases, compilers, debuggers)

– test Tools (software, discs, manuals etc)



• And just as importantly all items of Testware, such as: – Test Plans, Test Cases, Test Data, Conditions, Procedures, test environment

infrastructure components (hardware and operating software), test tools (purchased and bespoke) etc

• These must be:– Uniquely Identified

– Version controlled

– Tracked for changes

– Related to each other

– Related to other development items (functional specifications etc)

• Referencing to other documents allows for traceability

• The configuration management procedures and infrastructure (tools) should be chosen, documented and implemented during the Test Planning stage.



Risk and TestingRisk and Testing

Testing and Risk

Measuring Risk

Project Risks

Product Risks

Risk Based Testing in practice


Testing and RiskTesting and Risk

• Testing is an exercise in risk reduction

• Exhaustive testing is impossible

• As testers we are under constant time pressure

• Cannot reduce the risk of software failing to 0!

• Risk can be classified as two distinct types

– Project

– Product

Risk based testing


Project RiskProject Risk

• Project Risk: A risk related to management and control of the (test) project.

• Supplier Issues

– Contractual Issues

– Third party goes in liquidation or fails to deliver

• Organisational Issues

– Skills and Staff shortages

– Training and support issues

– Communication/Political Issues, e.g. between testers and other project teams

• Technical

– No or poor requirements

– Quality of the design or code

– Architectural solution under question

What is Project risk?


Product RiskProduct Risk

• Product risk: A risk directly related to the test object.

• “Buggy” software delivered

• Software does not meet its requirements

• Software delivered may harm the organisation, data or individual

• Poor software characteristics

– Functionality

– Security

– Reliability

– Usability

– Performance

• Product risks are a risk to the success of the project

What is Product risk?


Risk Based Testing in PracticeRisk Based Testing in Practice

Issue 1- We only have time to do half the testing we planned

If we have defined the risks of test items which we wish to test then we can focus on highest risk items first and deliver an application which fit for purpose

Issue 2 – we have 500 passes and 3 failures. Can we go live?

Again if we know the risk of the test items which make up 500 passes and 3 failures we can then make a decision based on the potential risk of failure of the items

left out standing or which have failed

• Risks can help decide where we should start testing or where we may need to more testing

• Risks also help us analyse our current state and through test monitoring we can determine if a system is ready to be implemented

• Risks can also drive the number of test levels and determine the techniques to use



“Prioritise so that when you stop testing you have made the best of the time you have available.”


Weymouth 2006



• Testing can support the identification of new risks – identified during test planning

• Testing is used to reduce the risk of an adverse effect occurring, or to reduce its impact

• Testing provides feedback about the residual risk- through measuring the effectiveness of critical defect removal and contingency plans

• In analysing, recording and managing the Product and Project risks the Test Manager is following well defined project management principles

• In a risk-based approach we can not only determine our test prioritisation but also the test techniques to use


Measuring RiskMeasuring Risk

RISK = Likelihood * Impact

Impact

Likelihood

High

HighLow

Low

High Impact and Likelihood– Needs to be addressed ASAP

Low Impact and Likelihood– Needs no action


C

Lik

elih

ood

Impact

Must TestCould Test

Won’t Test Should Test

D B

A

Statement Coverage 70%

Pair inspection

Informal Test specificationError Guessing

Formal Test Specification Statement Coverage 100%

EP/BVADecision TablesBranch coverage

Functional Risk Matrix - MoSCoW

Risk Based Testing in PracticeRisk Based Testing in PracticeL

ike

lihoo

d

Impact


Incident ManagementIncident Management

Definition

Basic principles

Benefits

Attributes of an Incident

Tracking and Analysis

Writing Good Incident Reports


Incident: Any event occurring that requires investigation

Or more simply put…

Any discrepancy between actual and expected results

Definition Definition

=expected actual!


Basic Principals

• All incidents should be raised

• Incident should be tracked from discovery through classification and correction to confirmation

• A formal incident management process and workflow is required to effectively manage incidents

• Incidents may be raised at ANY point in the lifecycle– Requirements review

– Coding

– Testing

• Incidents may be raised AGAINST any product in the lifecycle– Source Requirements

– System Designs

– Source Code

– Test Material

– Help or user guides


BenefitsBenefits

• Incidents:

– Provide developers and other parties with feedback about the problem to enable identification, isolation and correction as necessary

– Provide test leaders a means of tracking the quality of the system under test and the progress of the testing

– Provide ideas for test process improvement


Attributes of an IncidentAttributes of an Incident

• Date of issue, issuing organization, author, approvals and status

• Scope, Severity* and Priority* of the incident

• References, including the identity of the test case specification that revealed the problem

*severity: The degree of impact that a defect has on the development or operation of a component or system.

*priority: The level of (business) importance assigned to an item, e.g. defect.

Incident logging – Key attributes


• Expected and actual results

• Date the incident was discovered

• Identification or configuration item of the software or system

• Software or system life cycle process in which the incident was observed

• Description of the anomaly to enable resolution

• Degree of impact on stakeholder(s) interests

• Status of the incident (e.g. open, deferred, duplicate, waiting to be fixed, fixed awaiting confirmation test or closed)

• Conclusions and recommendations

• Global issues, such as other areas that may be affected by a change resulting from the incident

• Change history, e.g. of actions taken by project team members

Incident logging – Other attributes

Attributes of an IncidentAttributes of an Incident


• raised and logged

• prioritised and assigned

• under investigation

• fixed by developers (or technical writers)

• tested

• released to live environment

• resolved (closed)

• invalid incident report (closed)

• irreproducible problem (closed)

• enhancement (to be actioned next upgrade)

• on hold (deferred pending some other action)

Tracking and AnalysisTracking and Analysis

Tracking and analysis – Incident Status examples


• categorise incidents by type

• assign an appropriate priority according to incident severity

• report incidents to the appropriate management level

• determine incident causes and propose corrective measures

• ensure timely corrective action by reviewing incidents and tracking their clearance

• re- testing and reviews after changes have been made to the software (refer regression testing)

• review corrective measures to determine their effectiveness

• analyse deficiency trends to prevent non-conforming software

• N.B. Incident Review Boards often used to to oversee the reporting, analysis and timely resolution of incidents during the course of a project. Members from all interested parties, Project Manager, Test Manager, Lead developer, Customer, CM Manager etc

Tracking and analysis - steps

Tracking and AnalysisTracking and Analysis


Writing Good Incident ReportsWriting Good Incident Reports

• The most important part of defect report is the failure description

• It will provide the developer with all the information they need to repeat the incident and then fix it

• Badly written reports– Slow the project

– Provide bad metrics

– Delay fixes

• Write in short sentences which are to the point – accurate, complete and concise

• Contents of a failure description– Summary

– Steps to reproduce

– Isolation

What makes a good incident report



• Summary

– one or two lines which provide an overview of the incident and its severity and impact. Be sharp and to the point.

• Steps to reproduce

– provide the exact steps that were undertaken to create the incident. Be as concise as possible, but make sure you add EVERY step

– get these steps right makes it easier for the developers to reproduce. – it reduces the “it works on my machine” phenomenon

• Isolation

– is the incident repeatable, what particular factors effect the ability to reproduce the incident, For example – “I observed the error on the following platforms IE5, NS7”

Contents of a failure description



Summary

There were a number or errors on the add customer screen

Steps to reproduce

1. Opened the add customer screen

2. Entered a new customer

3. Pressed add

4. Got error message

Isolation

I tried on a few different branches and it worked on most of them

Bad Example



Summary

Error “cannot find object” (see attached screen shot) message was displayed when trying to add a new customer to the system using screen ADD_CUST.

Steps to reproduce

1.Opened the add customer screen using the menu

2.Entered a new customer (details are attached in spreadsheet)

3.Selected customer as “corporate”

4.Added to branch “Littleton”

5.Pressed add

6.Got error message “cannot find object”

Isolation

I tried on a few different branches and the system worked without issue when the branch was classified as open. The Incident only occurred when the branch was classified as closed

Good Example


Test Management - SummaryTest Management - SummaryWe learned about the Test organization

• The importance of independent testing

• The types of Independent test organisations

• The benefits and drawbacks of independent testing within an organization

• The different team members to be considered for the creation of a test team

• The tasks of typical Test Leader and Tester

We learned about Test planning and estimation

• The different levels and objectives of test planning

• The purpose and content of the Test Plan and what we need to consider when writing the Test Plan

• Difference between different estimation approaches: the metrics-based approach and the expert-based approach

• Test preparation and execution tasks that need planning

• The need for, and examples of, adequate exit criteria for specific test levels and groups of test cases


Test Management - SummaryTest Management - Summary

We learned about Test progress monitoring and control

• Why we need to Monitor testing and how, i.e. what metrics we gather

• Understanding and interpreting the metrics for Test Reporting, plus the contents of a Test Report

• Why we need Test Control

We learned about Configuration Management

• Why we need CM for testing and how it supports testing

• Typical Configuration items, including Testware


Test Management - SummaryTest Management - Summary

We learned about Risks and Testing

• What a Risk is

• The two main types of Risk - Project and Product

• That risks are determined by likelihood (of happening) and impact (harm resulting if it does happen)

• How we should employ a Risk Based Testing approach – benefits Testing and the Project

And finally we learned about Incident Management

• The definitions of Incidents and Defects

• The Benefits of Incident Management

• The Attributes of an Incident

• How we should track and Analyse Incidents

• And what makes a Good Incident Report

• EDS Internal

03-23-05Version 2.3

Tool Support for TestingTool Support for Testing



Types of Test tool

Effective Use of Tools – Benefits and Risks

Introducing a Tool into an Organisation

Summary


Types of Test ToolsTypes of Test Tools

Test Tool Classification

Tool Support for Management of Testing

Tool Support for Static Testing

Tool Support for Test Specification

Tool Support for Execution and Logging

Tool Support for Performance and Monitoring

Tool Support for Specific Application Areas

Tool Support using other tools


Types of Test ToolTypes of Test Tool

• Tools are classified according to the test activities they support (e.g. test management, static testing, etc.)

• Some tools may support multiple activities while others just one

• Tool vendors may provide a suite of (hopefully) integrated tools for the testing lifecycle. (Major vendors include Mercury, Compuware, Borland, etc.)

• Tools can be any piece of purchased or locally developed software

• Tools can be tremendously helpful in improving the efficiency and quality of testing…

• …but can also be a waste of time and effort

Test Tool Classification (1)



• Some tools may be intrusive – i.e. they affect the test outcome itself – this is the Probe Effect

– For example, some tools that measure the performance of a piece of code may insert code to count or time instructions – though very detailed information can be obtained, the timings will be affected simply because of the extra code

• Some tools offer greater support to developers rather than testers e.g., Static Analysers are more commonly used by developers

• Some tools offer greater support for different parts of the software development lifecycle (e.g. Unit Testing, System Testing)

• Computer Aided Software Test Tools (CAST Tools) is a common term used in testing circles

Test Tool Classification (2)



“Just like any carpenter, a tester needs a toolbox full of tools to get the job done… But

...like a good carpenter, a good tester is not defined by his tools, but by his skill at using them ”

Scott Loveland et al 2005




• Management tools can be applied across the entire software development lifecycle:

– during unit testing, acceptance testing, etc.

– and by all project staff - by managers, developers, testers, etc.

Tool Support for Management of Testing (1)



Test Management Tools:• Help Manage tests and activities

– Provide management and control of test documentation, test cases, specifications and results

– Support scheduling of tests, logging results, recording test environments, etc.

• Interface to other tools– Link to test automation tools to run tests automatically

– Link to defect tracking tools to allow cross referencing to incidents/defects

– Link to requirements management tools to cross reference tests and requirements

• Support Version control– Allow test cases, logs etc. to be baselined, “checked-out”, modified, retrieved

and controlled (often via another ‘linked’ tool)




Test Management Tools - continued• Allow Traceability

– Allow traceability to be recorded between test and requirements

– Allow traceability to associated defects/incidents

• Support Logging– Allow recording of results within the tool

– Have results linked to ‘useful’ information (who, when, on what hardware, etc.)

• Provide Analysis– Produce analysis reports on testing progress and test coverage

– Produce reports on tests passed and failed

– Produce reports on incidents raised

– Tools differ in their ease of use, accuracy of the information and portability of results for distribution




Requirements Management Tools

• Primary purpose is to store requirements (as simple text, as Use Cases, formal language, etc.)

• Generally provide a means of recording the priority of each requirement

• To support testing, such tools can:

– Verify and validate requirements (consistency checking, missing links to other requirements)

– Allow tests cases to be linked to the requirements

• Test Teams should review requirements early in the lifecycle to check for consistency, testability and to allow test cases to be constructed




Incident Management Tools

• Also known as defect tracking tools• Record defects/problems/incidents/anomalies with virtually any aspect of

the project• E.g. operational problems, testing failures, documentation errors.

Fortunately, it is not common to use them for reporting problems with people

• Widely used by all members of a project team – particularly testers• They support the prioritisation of incidents (e.g. High for incidents that must

be fixed in 4 hours through to Low for incidents that may never get fixed.)




Incident Management Tools – continued

• Allow incidents and resulting actions to be assigned to project members. E.g. to Investigate, to test, to cost, etc.

• Record current status and history of an incident. E.g. New, or Rejected, or Resolved, or Waiting for test, etc.

• Some generate analysis reports on trends in incident creation, resolution, effort recorded, etc.

• Most can be tailored to specific project needs e.g. to define project specific states or priorities, to specify the layout of information, to construct predefined reports, etc.




Configuration Management Tools

• Allow software baselines to be kept and managed (See notes for background information)

• Not strictly testing tools, but are heavily used by test teams to:

– obtain specific version of software (code, documents, etc.)

– maintain their own test material

• Traceability between testware and software can be achieved by applying build/release labels to corresponding items

• Useful (often essential) when managing different version of a system for different target environments or different operational releases




Review Process support tools

• During a review:

– Documents are issued, people are nominated, roles are assigned

– Templates and guidelines are distributed

– Feedback is produced, times are recorded, people are ‘nagged’ etc.

• Tools support these activities by storing the documents, providing communication mechanisms, recording metrics, issue reminders, etc.

• Some support on-line reviews where people are geographically dispersed. (Within EDS, eRoom is used to support online reviews)

• Many companies (including EDS) use tailored Spreadsheets to record comments and generate review metrics

Tool Support for Static Testing (1)



Static analysis Tools

• Static analysis tools provide information about the quality of the software – by examining the code but not executing the code.

• Static Testing can often be carried out early in the development lifecycle – thus defects are detected sooner and (generally) more cheaply

• Static analysis tools usually give objective measurements of various characteristics of the software, such as the complexity measure

• They are used by developers, testers and quality assurance personnel

• Code defects detected include unused code, unused variables, coding standard violations, etc.

• Vendors include McCabe, Segue and many more.




Modelling tools

• Projects may have analysis models, design models, data models, state transition models, etc….

• These ‘models’ of the system are checked for defects

– For example, a state transition model may have states with no entry point or exit point, etc.

• Tools will require the model to be in a format they can ‘read’ - Perhaps as a formal language or as part of the tool itself

• Some tools actually produce test cases from the models

• These tools allow testing to take place early - defects are identified sooner




Test Design Tools• Test design tools generate some or all of the following automatically

– Test inputs (i.e. the test data) this would be data from external system or user

– Test cases

– Expected results – i.e. may use a “test oracle”

• They use requirements, analysis models, data models, state models, code

• Tools will require the model to be in a format they can ‘read’ - Perhaps as a formal language or as part of the tool itself

• The tools either have built in or configurable test generation criteria

• They may look for input boundary conditions, state transitions, paths through the system, etc.

Tool Support for Test Specification (1)



Test Design Tools - continued

• Their value is from:

– Ensuring thoroughness of test coverage by generating independent tests and data

– The time saved in test generation

• Some Test design tools also generate test stubs and templates from models and code




Test data preparation tools• Enable data to be selected from existing databases, files or captured

transmissions

• Enable data to be created, generated, manipulated and edited for use in tests

• The most sophisticated tools can deal with a range of file and database formats

• The application under test is often used to ‘Drive’ the data into databases, etc.

• They have the benefit of generating anonymous data for data protection and security purposes.

– i.e. They can generate data in the right format, with the right bounds, but isn’t actually live/real data




Test execution tools

• Provide capture, replay and comparison facilities

• Tools simulate mouse movement, button clicks and keyboard inputs

• Can recognise GUI objects and states for later comparison – windows, fields, buttons, bit maps, etc.

• Test scripts are normally captured in a programmable script language (VB, C, Java, etc.)

• Additional code can be added to the script for data manipulations, verifications, iterations, etc.

• When executed, tests run automatically, following the recorded/ programmed script using the defined input data

• Compare results with the predefined expected values and record the logs in the tool (i.e. they use a Comparator).

• A test Comparator may use a test oracle , especially if it is automated.

Tool Support for Execution and logging (1)



Test execution tools - continued

• Generating automated scripts can be time consuming, challenging and fun

• They are best used for tests which will be repeated many times (these tools are most often used to automate regression tests)

• Using the Capture-replay facilities of such tools, testers can record their actions – which can be very useful when failures occur. The complete history of the user actions can be replayed and studied.

• Can be used to replicate Users, i.e. multi-user testing.




Test harness/unit test framework tools

• test harnesses, drivers and stubs simulate other parts of the system or other connecting systems (or people)

• They are used to:

– Simulate other components that are not yet available – so testing can continue

– Simulate other components to allow detailed, controllable testing of a specific component

– Simulate events that may be difficult (even dangerous) to produce with the whole system (e.g. database failures or nuclear meltdown)

• Commercially available tools exist, but custom-written programs also fall into this category

• Simulators also come under this category




Test harness/unit test framework tools - continued

• Test harnesses inject data into a component and capture outputs from the component

• Comparisons with previous runs and pre-defined expected outcomes can be performed

• Some frameworks are designed for developers for unit/component testing -these are unit test frameworks (e.g. JUnit, HTTPUnit)




Test comparators

• Comparison tools are used to detect differences between actual results and expected results

• Standalone comparison tools normally deal with a range of file or database formats

• Test execution tools usually have built-in comparators that deal with character screens, GUI objects or bitmap images

• These tools often have filtering or masking capabilities - they can ‘ignore’ rows or columns of data or areas on screens (e.g. volatile date fields)




Coverage measurement tools

• These tools are dynamic test tools and allow test coverage to be monitored

• Most commercial tools are designed for code coverage i.e. how much of the code (statements, branches) has been executed

• Provide objective measures of test coverage when tests are executed

• Some tools add extra code to the source code (called instrumentation) to allow detailed information to be obtained – but recall the Probe Effect

• After execution, the log file is analysed and coverage statistics generated

• Statistics are provided on the most common coverage measures such as statement or branch coverage




Security tools• With the rise of the internet, the need for preventing malicious attacks on

systems has increased

• Tools have emerged which test a systems vulnerabilities to attacks

• Virus scanners monitor for unexpected arrival of damaging code (via email, embedded in web pages or documents, etc.)

• Denial of service attacks– Tools examine code and attempt to cause a failure by injecting rogue data.

– They may look for weaknesses in code (in HTML, .NET, etc.) deliberately trying to damage the service

• Penetration testing – trying to break in. Tools will look for security loopholes, monitor traffic, or password weaknesses, etc. and attempt to gain access to machines or services




Dynamic Analysis tools

• These are used to find defects that are apparent only when the software is actually running (see also Static Analysis Tools)

• These tools are most commonly used to: -

– identify unassigned pointers

– monitor the allocation, use and de-allocation of memory to flag memory leaks

• These tools are generally used by developers

Tool Support for Performance & monitoring (1)



Performance testing tools

• Performance test tools have two main facilities - load generation measurement and test transaction measurement

• Load generation can simulate either multiple users or high volumes of input data

• Load generation is done either by driving the application using its user interface or by test drivers

• Records of the numbers of transactions executed are logged

• Response times are taken and logged for selected transactions

• Performance testing tools normally provide reports based on test logs and graphs of load against response times

• Tools in this category are also called load test tools or stress test tools




Monitoring tools

• These tools are not strictly test tools, but enable failures or potential failures to be detected

• Monitoring tools monitor the systems memory, CPU usage, file spaces, network traffic, disk I/O, current processes running, database optimisation (e.g. monitoring embedded into products such as Oracle) etc.

• They can be ‘tuned’ to report potential failures – e.g. file space 80% full, process XYZ should always be running, etc.




• Tools exist which are specialised for a specific task, environment or application

• For example:

– Harnesses, monitoring tools, etc for embedded systems – where their specialised hardware and application requires specialised tools

– hyperlink testing tools (Spiders) are used to check that no broken hyperlinks are present on a web site

– Tools for a specific operating system (Windows, UNIX, etc.)

– Tools for a specific language (JAVA, C++, .Net, etc)

– Performance tools specifically for WEB based user interfaces

Tool Support for Specific Application areas



• Many other tools are used to support the test effort.

• For example:

– Word processing tools – widely used for producing test plans, test reports, to record test cases, etc.

– Spreadsheets – also widely used for test cases, cross-referencing, analysis of data, reporting, etc.

– Database interrogation tools (e.g. SQL, Toad, Raptor) – database interrogation is performed to validate the contents of tables. Simple SQL statements can be used or more sophisticated tools (Toad, Raptor, etc. which allow the ‘novice’ to interrogate a database)

– Operating system utilities and scripts to monitor resources, generate data, run tests

– Code debugging tools – to step through code, monitor variables, etc.

Tool Support using other tools


Effective Use of Tools – benefits and risksEffective Use of Tools – benefits and risks

• Some tools bring quick and easy benefit to the test effort, but…

• Some tools require considerable investment in time and thought to realise their benefits

• Don’t just look at the adverts or listen to the salesman

Potential Benefits and risks – all tools (1)



Potential Benefits of using tools

• Repetitive work reduced – “Relieve the boredom”

– re-running the same test many times can be extremely boring and error prone

– Tools can reduce the time we spend on repetitive tasks

• Consistency and repeatability – “Errare Humanum Est” (to Err is Human)

– Test identified by a tool or run by a tool can be more repeatable and consistent. Tools do what they are told

• Objective assessment – tools have no “axe to grind”

– they are objective with their assessments and results

• Ease of access to information

– tools can give easy access to test information stored and present it in multiple formats for distribution

• Automated tests can be run out of hours




Risks of using tools - every silver lining has a cloud

• Unrealistic expectation - Don’t just look at the adverts or listen to the salesman. Tools may be rich in features – but are they useful to your test effort?

• Underestimating how much effort/money is required to purchase, install, get trained and simply get started with the tool

• Underestimating how much effort is required to gain experience so that benefits can be realised

• Underestimating the effort to incorporate into the test process

• Underestimating the effort required to maintain the assets (automated scripts may need to change for every change in the data or user interface)

• Over-reliance – not considering other tests that need to be found or test that need to be run




“Selection of the tool affects the effectiveness and efficiency of testing. .

The technique for hammering a nail into a piece of wood is well understood…

… if the wrong hammer is selected the entire process can be inefficient….”

William E. Perry 2000




Test Execution tools

• Scripts for such tools are generated in a scripting language (usually VB, C, Java)

• Scripts can be very time-consuming to produce and difficult for non programmers to maintain - which could delay significant testing while creating automated tests

• Scripts do exactly what they are told – and nothing more. A simple pop up from another application on the desktop could halt the script

• Two approaches are used to assist with script generation:

– Data-driven approach – script is produced to read test data from (typically) a spreadsheet. Testers run tests with different data by manipulating the spreadsheet and not the script

– Key-word driven approach – scripts are produced using building blocks – each given a key-word. E.g. “Logon”, “Add_User”, “Delete_User”. Testers create tests by picking the ‘blocks’ they need

Special Considerations for some tools (1)



Performance Testing Tools

• Requires specialists to design, execute and analyse the results

• Often requires detailed understanding of how the system is constructed (databases, networks, etc.)

• Often requires detailed understanding of expected system usage profiles

• Need to be re-run when system ‘tuning’ takes place




Static Analysis Tools

• Code Static Analysers examine code and reports on defects and ‘bad practice’

• Are best used at the start of coding to ensure problems are removed quickly

• But…

• Many systems contain automatically generated code, which often causes static analysers to generate many reports

• Applying these tools to legacy code can produce a flood of unhelpful messages

• Get the tool to filter out unwanted messages and so target the required code




Test Management Tools

• Need to interface with other tools. E.g to import and export information from word processors and spreadsheets

• Must help your test process – not hinder progress

• Must ensure the information they hold is readily available for viewing and distribution

• Must produce reports easily



Introducing a Tool into an OrganisationIntroducing a Tool into an Organisation

• Organisational Maturity –

– Tools should help the projects test effort and should fit into the culture

– Spending time introducing automated execution tools for rapid reaction or prototype projects may be of no value

• Evaluation of product –

– Produce weighted Acceptance Criteria

– Determine test tool options, prices and conditions

– Produce a Return on Investment (ROI)

– Investigate the market to see what products are available

– Produce a Short List of suppliers

– Conduct evaluation tests against application(s) with at least two suppliers and score results

Principles to follow (1)



• Proof-of-concept – before rolling out the tool across the organisation, “test it out” on a manageable, resourced, pilot project.

– If it fails – the impact will be minimised

– If it succeeds – greater experience will be available and organisational roll out can be better supported

• Evaluate the vendor

– Discuss contractual issues – company stability, licensing and maintenance costs, usage restrictions

– Evaluate training courses, technical support and consultancy

– Review the tools development roadmap – is it going in the right direction for you

• Internal Coaching

– Identify staff to be trained

– Who will provide internal support for the tools installation and usage




• Other questions to ask:

– What protocols, languages, networks, hardware are being used now and in future?

– What are the budget and timescale for delivery?

– What is the problem to be solved?

– Are there alternative solutions – other than tool purchase?

– Are there any specific tool features and characteristics?

– Do you need a project specific tool, or is this for the organisation?

– But avoid too many tools of the same type

- tools are not just for Christmas




“Identifying, investigating, and evaluating tools are important responsibilities of a test leader…

… But it’s important to recognize how much time and effort is spent doing so.”

Scott Loveland et al 2005




• Learn about the tool

– No training course or brochure will give enough information about how the tool actually works in your environment.

– It is vital to learn about the tool before rolling it out for wider use

• Is the tool a good fit?

– Will the organisations processes accommodate this tool?

– What organisational processes need to be updated?

– Can it be integrated with other tools sets for improved effectiveness?

• Generate a best practice guide

– create guides and work instructions in the use of the tools

• Benefits?

– understand and measure as objectively as possible the benefits to be gained

– will it be worth the effort?

Objectives of the Pilot project



• Incremental roll out

– Avoids a sudden increase in demands for extra support and training

– Allows experience to be gained and passed on ensuring a more successful roll out

– If problems are encountered, there is less impact if the roll out is delayed or aborted

• Adapt processes

– Processes may benefit from changes because of the tool

– For example, when and how regression testing is done (with automation), how test cases are developed and recorded, etc.

• Training

– To avoid tools becoming shelfware new users will need local training and support

– Some users will be easily deterred by new tools

Success factors for tool roll-out (1)



• Produce Guidelines

– Tools may come with extensive user guides but it is useful to supplement them with local conventions, FAQs, standards, etc.

– make the guidelines easily accessible

• Learn lessons and monitor usage

– As tool usage increases capture users experience for sharing and process improvement. Consider:

• Web based forums

• Email distribution groups

• Discussion groups

• Nominated tool authorities to approve changes

• Regular assessment of benefits vs costs




• Other success factors:

– Maintain close relationship with tool vendor – to understand their future plans, to get quick access to information

– Monitor test tool market for new or improved products

– Ongoing research into how to better utilise test tools

– Use consultants to resolve short term or difficult issues where there is no gain from wasting resource time on complex one-off problems



EDS Application Engineering Toolkit (AET)EDS Application Engineering Toolkit (AET)

• The Applications Engineering Tools team evaluates, recommends and pilots standard Enterprise Application Development Life Cycle tools, utilities and frameworks to enable the delivery of service offerings within the global applications engineering organization

• AET Objectives are to: -

– Improve applications development sustainable productivity and reduce rework through the deployment of integrated toolkit and framework recommendations

– Decrease operating cost and improve functionality for dollar spent through implementation of application software and database product replacement recommendations

• http://www.gsms-ea.eds.com/Tools/index.html

http://www.gsms-ea.eds.com/Tools/index.html


Collateral on ToolsCollateral on Tools

• QA Forums - http://qaforums.com/

• StickyMinds - http://www.stickyminds.com/

• Automation Junkies - http://www.automationjunkies.com/

• Software Test & Performance - http://www.stpmag.com/

External Sites (non supplier based)

http://qaforums.com/



http://www.stickyminds.com/



http://www.automationjunkies.com/



http://www.stpmag.com/





• Firstly, we looked at types of test tool and how they are classified…

Summary (1)



Summary (2)

Test management toolsTest management

Incident management

Requirements management

Configuration management

Static testing toolsStatic analysis

Modelling

Review process support

Test specification tools Test design Test data preparation

Test execution and logging tools

Test execution

Test comparators

Security

Test harnesses

Coverage measurement

Performance, monitoring toolsDynamic analysis

Monitoring

Performance/load/stress

Application specific toolsEmbedded systems

Language specific

Web testing

Other toolsWord Processors

Spreadsheets

Operating system utilities

SQL, Code debugging



• Then we looked at the effective use of tools:

– Benefits of tools:

• Save time, thorough testing, reduce repetitive tasks, etc.

– And the risks:

• Underestimating time and effort, Over-reliance, etc.

– Special considerations for:

• Test execution tools, performance tools, static analysis tools and test management tools

• And finally, we looked at how to introduce a tool:

– Main principles (evaluation, assessments, etc.)

– Pilot project objectives

– Success factors

Summary (3)


IndexIndexThe following slides provides a quick reference guide to some of the key terms used in the course, to aid revision. Each term is hyperlinked and has slide numbers plus associated ISQTB Syllabus section numbers. Remember to use the Glossary of terms for exact definitions.

Term Slide(s) Topic Syllabus reference

Acceptance Testing 116 Testing Throughout the S/W Lifecycle 2.2.4

Black Box Testing 197 Test Design Techniques 4.2

Boundary Value Analysis 211 Test Design Techniques 4.3.2

Component Integration Testing 85 Testing Throughout the S/W Lifecycle 2.2.2

Component Testing 80 Testing Throughout the S/W Lifecycle 2.2.1

Configuration Management 282 Test Management 5.4

Decision Tables 214 Test Design Techniques 4.3.3

Decision Testing 229 Test Design Techniques 4.4.2

Equivalence Partitioning 204 Test Design Techniques 4.3.1

Error 10 Fundamentals of Testing 1.1.2

Error Guessing 239 Test Design Techniques 4.5

Exit criteria 261 Test Management 5.2.3


IndexIndex


Exploratory Testing 241 Test Design Techniques 4.5

Failure 12 Fundamentals of Testing 1.1.2

Fault 10 Fundamentals of Testing 1.1.2

Formal Review Process 166 Static Techniques 3.2.1

Functional Testing 131 Testing Throughout the S/W Lifecycle 2.3.1

Fundamental Test process 44 Fundamentals of Testing 1.4

General Principles of Testing 34 Fundamentals of Testing 1.3

Incident Management 295 Test Management 5.6

Independent Testing 248 Test Management 5.1.1

Informal Review 162 Static Techniques 3.2.3

Inspection 165 Static Techniques 3.2.3

Maintenance Testing 140 Testing Throughout the S/W Lifecycle 2.4

Non- Functional Testing 132 Testing Throughout the S/W Lifecycle 2.3.2

Product Risk 289 Test Management 5.5.2

Project Risk 288 Testing Throughout the S/W Lifecycle 5.5.1

Regression Testing 136 Testing Throughout the S/W Lifecycle 2.3.4

Re-testing (Confirmation) 135 Testing Throughout the S/W Lifecycle 2.3.4


IndexIndex


Risk Based Testing 287 Static Techniques 5.5

State Transition Testing 217 Test Design Techniques 4.3.4

Statement Testing 225 Test Design Techniques 4.4.1

Static analysis 178 Static Testing 3.3

Structural Testing 133 Testing Throughout the S/W Lifecycle 2.3.3

System Integration Testing 111 Testing Throughout the S/W Lifecycle 2.2.2

System Testing 95 Testing Throughout the S/W Lifecycle 2.2.3

Technical Review 164 Static Techniques 3.2.3

Test Case 191 Test Design Techniques 4.1

Test Condition 190 Test Design Techniques 4.1

Test Estimation 263 Test Management 5.2.4

Test Execution Schedule 194 Test Design Techniques 4.1

Test Monitoring/Control 272 Test Management 5.3.1

Test Objectives 27 Test Management 1.2

Test Planning 256 Test Management 5.2.1

Test Procedure 193 Test Design Techniques 4.1


IndexIndex


Test Tool types 312 Test Tools 6.1

Use Case Testing 222 Test Design Techniques 4.3.5

V-model 72 Testing Throughout the S/W Lifecycle 2.1.1

Walkthrough 163 Static Techniques 3.2.3

White Box Testing 199 Test Design Techniques 4.2


Statement of ConfidentialityStatement of Confidentiality

The information contained in this document is proprietary to EDS. EDS submits this document with the understanding that it will be held in strict confidence and will not be disclosed, duplicated or used, in whole or in part, for any purpose other than the evaluation of EDS’ qualifications, without the prior written consent of EDS.

EDS is a registered mark and the EDS logo is a trademark of Electronic Data Systems Corporation.

EDS is an equal opportunity employer and values the diversity of its people.Copyright 2006 Electronic Data Systems Corporation. All rights reserved.

[email protected] Presentation and Course owner Paul Weymouth, UKIA Testing ADU

EDS and the EDS logo are registered trademarks of Electronic Data Systems Corporation. EDS is an equal opportunity employer and values the diversity of its people. © 2005 Electronic Data Systems Corporation. All rights reserved.

Presentation and Course contributors:

•Paul Weymouth, Testing Architect UKIA Testing ADU

•Mark Otter, Senior Test Consultant Australia South ADU

•Dave Broughton, Testing Architect UKIA Testing ADU

30 Sept 2005Version 2.3

mailto:[email protected]

istqb tfc v2.3

Documents

fundamentals of testing

eds internal version

eds internal areas

uk testing adu slide

eds exceptions

eds internal objectives

external testing guru

testing related activities