comp8130 and 4130adrian marshall 8130 and 4130 test management adrian marshall

COMP8130 and 4130 Adrian Marshall

8130 and 4130

Test Management

Adrian Marshall


Overview

• Principal Testing Activities• Testing Problems• Scoping, Scheduling and Estimating• Planning• Documenting• Controlling• Test Teams (and independence)


Principal Testing Activities

Planning• scope, approach, resources, budget, documentation set, roles &

responsibilities, schedule

Preparation• test documents, tools, environments, training

Execution• developers, systems integrators, users, external specialists, Independent

V&V agents

Reporting• informal, formal, automated

• defects, progress, resource usage, issues & risks….


Testing Problems

Modern software systems present the following potential problems to testing:

•Large, complex and expensive distributed processing / multi-tier systems are common•Lack of a single end-to-end solution development owner•Variability of component maturity and quality•Variability of supplier capability and workmanship•Significant component interface and integration problems


Testing Problems


•Large user populations, and large numbers of concurrent users•Unpredictable user loads which are often driven by uncontrollable circumstances•Enterprise wide implementation of software systems creates critical business dependency on systems availability ….


Testing Problems


•Significant component interface and integration problems•Large user populations, and large numbers of concurrent users•Unpredictable user loads which are often driven by uncontrollable circumstances•Enterprise wide implementation of software systems creates critical business dependency on systems availability ….


Testing Problems

Problems can occur in:•Quality focus;•Test scope;•Test priorities;•Test budget;•Test schedule;•Test skills and experience;•Test documentation;•Communications.


Scoping, Scheduling and Estimating

How do we scope the testing?• Understand the system to be tested

(requirements)

• Understand the risks

• Understand the development process

• Understand the technology

• Understand the business objectives



Where do we get our requirements from?

Operational requirements documents• User feedback from prototypes, field trials, site

visits• Existing systems and their documentation

Customers:• System specification(s)• Statement of work, • Contract• MIL, ISO or AS Standards• Customer Furnished Information




Specifications:

• Interface Specifications

• Product Requirements Specifications,

• Marketing Specification

• Drawings

• Technical Specifications and standards




Corporate:

• The bid

• Marketing Specification

• Company and project policies,

• standards and procedures

• Existing design



Analysing the risks

• Hazard identification• Business risks

• System risks

• Technology risks

• Management risks

• Risk sizing

• Risk management / mitigation



Understand the development process (and influence it?)

• Incremental

• Iterative

• Use Case definitions

• Integration methods

• Phasing of builds



Understand the technology

(examples):

• PKI

• Embedded systems

• Windows / Unix

• Smart-card

• EFTPOS

• jpeg



Understand the business objectives

(examples):

• Time to market critical

• Phased delivery

• Low price, high volume sales

• Security, confidentiality, IP protection



Scoping.

Finally…..

…….determine the required tasks to fulfill the scope…..

Inspections, unit, integration, system, acceptance, performance, reporting



Scoping exercise:Consider requirements, risks,

development process, technology and business objectives for the development of:

a. R&D program for biometric authentication;

b. Game based on the latest movie;c. Fuel management system for the

A380



Estimating

• Understand the scope

• Understand the size of the system (may use COTS products) and the size of the development

• Understand the complexity of the system

• Historic data for similar testing scope

• Gut feel

• Rule of thumb



Estimating - Useful numbers:

Overall testing effort (all activities) = 30% of the Test planning = 15% of total test effort

Designing and documenting tests cases = 10% of total test effort

Test procedures and test execution = 70% of total test effort

Test reporting = 5% of total test effort



Estimating

• Evaluate estimates• Record actual effort (test metrics)

• Don’t be too granular (testers aren’t accountants or lawyers)

• Refine estimates

• Refine schedule



Scheduling• Identify the sequence of events

• Identify the critical path

• Communicate the draft schedule Development team Project manager

• Refine the schedule (communicate and changes to the risk profile)



Scheduling and estimating exercise (small groups):• Three month schedule from start of

development to release

• First build due at the end of the second month

• 12 developers working fulltime

• Estimate the effort, size of the test team, and present a simple schedule


Planning

Building the test plan:

• Scope, Schedule and Estimate

• Explain the test execution strategy

• Allocate roles and responsibilities

• Outline the testing schedule

• Identify test resource requirements

• List assumptions, issues and risks

Approving the test plan

• Have appropriate stakeholders authorise / sign the plan….


Planning

• Planning should be done very early - be part of the project bid / project initiation team

• Obtain clear solution success / acceptance / quality criteria to enable identification of test focus areas

• Identify & evaluate risks and create a testing strategy which helps to manage them

• Produce a professional and defendable Test WBS

• Be appropriately independent

• Act as a support service not an authority figure

• Keep testing team and ‘customers’ informed

• Be involved across the project lifecycle….


Documenting

Why do we document?

What do we document?

How do we document?

Master Test PlanEntry and exit criteriaUnit level testing documentationDocumentation for subsequent phasesPlans are not static… they must be constantly evaluatedTest CasesTest ProceduresTest Reports


Controlling

What is needed to control the activities?

Metrics• Rate of test design

• Rate of execution

• Product quality

• Product stability

Reporting


Test resources

Test Teams

Test Environments

Test Tools


Test resources

Test Teams

• Independence

• Identify skills / experience

• Hiring / contracting

• Motivating / retaining


Test resources

Test Environments

• Procure

• Develop

• Control/manage


Test resources

Test Tools

• Evaluate

• Procure

• Develop

• Validate/verify

• Training

comp8130 and 4130adrian marshall 8130 and 4130 test management adrian marshall

Documents

adrian marshall scoping

problems scoping

systems availability

independence slide

following potential

systems integrators

test teams

site visits existing