managing testing

8/14/2019 Managing Testing

1/39

1

Managing testing

performance of tea ms, inc lud ingoutsourc ed testing

From the authors of:
http://search.barnesandnoble.com/Six-Sigma-for-the-Next-Millennium/Kim-H-Pries/e/9780873896566/?itm=1


2/39

2

What you will learn:

In advance of testingRole of Reviews

Simulation

Testing Scope

Software

Hardware

Testing outsourcing

Risks and mitigation


3/39

3

Technical Reviews

Specification reviews

Code reviews

Walk through

Test plan reviews

Test case reviews


4/39

4

Test Deliverables andMetrics

Integration into development process

Comprehensive to the development process

Project phases

Test Evaluation Master Plan

Design for testability


5/39

5

Testing Preparation

Understand REAL demands on a Product (PD&D article)

www.pddnet.com/ sc ripts/ ShowPR~RID~23252.asp

www.pddnet.com/ Sc ripts/ ShowPR~RID~23304.asp

Human expertise

Tools

Signal (features)

Noise (external influences)

Communications
http://www.pddnet.com/scripts/ShowPR~RID~23252.asphttp://www.pddnet.com/Scripts/ShowPR~RID~23304.asphttp://www.pddnet.com/Scripts/ShowPR~RID~23304.asphttp://www.pddnet.com/scripts/ShowPR~RID~23252.asp


6/39

6

Simulation and test(Evaluation)

Support or refute design or concept direction

Reduces reliance or hardware and softwareavailability (dependence)


7/39

7

Technical Reviews

Specification reviews

Code reviews

Walk through

Test plan reviews

Test case reviews


8/39

8

Test metrics

Incoming fault rates (faults per sw/hw release)

Categories of faults found

Specification inconsistencies

Quality perception

Severe impact

Time to fault closure (time to correct faults)

Reliability


9/39

9

Test metrics (continued)

Faults found not during testing

Testability

Specifications written able to generate test criteria

System or component designed to be able to test

Performance to invalid inputs


10/39

10

Test statistics

Model defect arrivalrates againstRayleigh distribution

Establish a defectcontainment matrixto see if defects

trapped betweenreleases


11/39

11

Test ArtifactsLab Testing

Sample parts

Test plan

Scrap

Test report


12/39

12

Test ArtifactsProduction Line Testing

Production parts

Test plan

Scrap

Test results stored

Parts passed


13/39

13

Product Requirements

Compliance testing (against specs)

Extreme testing (challenge the product)

Combinatorial testing (cover the product)

Stochastic testing (find new ways totorture the product)


14/39

14

Span of Tests

Softwaremodule test (unexpected inputs)

build tests

regression testing

Component test

System

Sub-system testSystem Integration test

System live test

Production test


15/39

15

Span of Tests (continued)Quality Council of Indiana

Static analysis

Static tests

Structural Analysis

Dynamic tests

Mathematical evaluation

Simulation

Prototyping and simulation


16/39

16

White box testing

Intimate knowledge of outputs, inputs,and the box

In short, we should know how the

transformation of inputsoutputs occurs

May be modeled as sets of inputs andoutputs inside the box


17/39

17

Black box testing

We know the inputs

We know the outputs

The box is a mystery

The goal is to characterize the relationship ofthe outputs to the inputs

Often a candidate for a combinatorialapproach


18/39

18

Test Deliverables andMetrics

Integration into development process

Design for testability

Clear pass fail criteria (function or physical)


19/39

19

Production test equipment

Purpose is to remove failing material,NOT to pass good material

High degree of automation

May use specialized equipment

In-circuit testersCustomized to a specific product


20/39

20

Outsourcing

Objectives with outsourcing test (what to outsource)

Risks & Risk Mitigation


21/39

21

Test reporting

System qualitiesAccessibility by team

Traceability

Categorization

Assigned responsibilities

Statistical analysisSpeed! !


22/39

22

Review test results

Hardware issues

Software issues

Corrective actions


23/39

23

Supplier Integration intodevelopment

Supplier (contractor) requirements

Supplier (contractor) participation

Supplier (contractor) audit


24/39

24

Planning in- or out-sourcing

Currently using a scrum approach totesting

By product backlog

Daily scrum meeting

Burn down charts


25/39

25

Why does this work?

Microtasking breaks larger items down to atomiclevel

Unwise to decompose further

Completion is BooleanYes or no

Rollup can provide a % for upstream

managers


26/39

26

Microtasking

Like eating an elephantone bite at a time!

Immediate reinforcement to testers

Feeling of accomplishment

Measurable completion

Employees not overwhelmed at any given task


27/39

27

Reviews

Sprint review equivalent to a monthly or fortnightly in-process review (IPR)

Basically, good systems engineering

Out-source can use technology

Example: Microsoft Live Meeting

Example: eroom


28/39

28

Month versus fortnight

Fortnightly sprints increase reviewburden but also improve feedback andcontrol

A calendar month may be too long aperiod, especially during periods of

great change

Judgment call


29/39

29

Objectives: with outsourcingtest (what to outsource)

Objectives with outsourcing testwhat to outsource

Outsource

What we cannot test with our ownresources

What an outside lab can do better

Unusual tests

Risks & Risk Mitigation


30/39

30

Outsourcing: Risks & RiskMitigation

RiskDont know product

Dont know how to use fixtures and support

tools

May not understand special instructions


31/39

31


MitigationUp front review of outsource facility

Review of outsourced:

test plans Test fixtures

Test cases (test execution specifics)

Test report demands identified

Audit conduct the a few of the tests in housethe answers should be the same as the

outsourced test activities


32/39

32


Send an engineer to test site (extra cost)

Provide fixture (or test cases)

Frequent test status review

Clear test metric identification and tracking

High Reliability and Confidence interval Large sample size


33/39

33

Outsourcing: Risk Mitigation

Clear and detailed Responsibility andAuthority Matrix

Communications plan

Use of test standards

Clear pass fail criterion

Physical

Functional


34/39

34

Outsourcing: Pros / Cons

Pros

Increase capacity instantly

Often, special expertise

Cons

Cost

Product-related issuesCommunications and Logistics

Expertise retention


35/39

35

Outsourcing: Reliability

Use accredited labs

Solicit samples of reports

Visit site


36/39

36

Test configurationmanagement

Product baseline

Product change management

Product configuration (CMP)

Test configurations

Configuration audit


37/39

37

Test Planning

Project integrated Test plan

Project management responsibility

Test Evaluation Master Plan

Product release schedule (revision andfunctional content)


38/39

38

Human resources

Manager

% Engineers

More complex problems

Test cases & procedures

Can write code for test machines

% Technicians

Basic test execution

Simpler problems


39/39

39

Additional information

Kim Pries: [email protected]

Jon Quig ley:jonmquig [email protected] om
mailto:[email protected]:[email protected]:[email protected]:[email protected]

managing testing

Documents