Q

Best Practices in System Testing

Mahesh KurubaDr.Gargi Keeni

International Symposium on Future Software Technology, Xi’an

October 2002

QObjective

To emphasize the significance of SystemTesting and its Best Practices to improvethe System Test Processes

QAgenda

Introduction 5 mins

System Testing 10 mins

Best Practices 40 mins

Challenges 5 mins

Quiz 30 mins

Test Management Metrics

30 mins

Exercise 30 mins

QWhat does a defect mean to the

customer

American Airlines - $20,000 per minute for outage

Sabre Reservation system - $50,000,000

Denver airport - $1,100,000 per day

Source:Ed Bryce, “Failure is not an Option”, http://www.stickyminds.com/docs_index/XDD3316filelistfilename1.pdf

Q What does Down time mean to the customer

$1,000 to $27,000 per minute of down time

ESPN - $7,800,000 for 3 days

Amazon.Com - $400,000 for 6 hours

Egghead.com - $280,000 for 2 days

American Airlines

Sabre reservation system 1 min - $20,000 12 hrs 1989, 5 hrs in

1994

Source:Ed Bryce, “Failure is not an Option”, http://www.stickyminds.com/docs_index/XDD3316filelistfilename1.pdf

Q

Market / Customer perspective– Reputation– Market share

Organization perspective– Productivity loss– Schedule slippage– Loss in Profit Margin

What does a defect mean to the Software Development

Organization

QCost of Uncovering Defect

Phase CostRequirements $63.75Integration/System Test $750-

$3,000Production $10 K -

$140 K

QCost of Fixing defect

Phase in which defect Found Cost ratio

Requirements 1Design 3-6Coding 10Unit/ Integration Testing 15-40System testing/ Acceptance testing 30-70Production 40-1000

QROI Analysis

Source: Rex Black, The Cost of Software Quality,Investing in Software Testing Series, Part 1www.stickyminds.com

Q

System Testing

QVerification and Validation

Review Inspection Unit testing Integration testing System testing

Requirements Design CodingUnit

TestingIntegration

Testing

Verification Verification

Verification

Validation

Validation

QWhy System Testing?

Scope of system testing e-Security testing Compatibility testing Performance testing Usability testing Disaster Recovery testing

QV-Model

Requirements

Design

Coding Unit Testing

Integration Testing

System Testing

QSystem Test Strategy

Objective– What type of project?– What type of software?– What is the technical environment?– What is the project scope– What are the success factors?– How critical is the system to the

organization?– What is the required Quality level?

QSystem Testing Process

Testability Assessment

Test Automation Assessment

Performance Assessment

Requirements Traceability Matrix

Test Strategy

Test Plan

Test Cases

Testing

Test Results and Analysis

QSystem Test Activities

    REVIEW 

Test Cases

Review

Testing

Requirements

Test Results

Design Coding/Implementation

System Testing

Review

Design DocRequirements Doc

Test Strategy

Test Plan

Testability Assessment

Test Automation Assessment

Q

Best Practices

Q

Requirements traceability Design – to – Test Design of Test Cases Reviews Production data Traceability of defects Train the Testers

Best Practices…

Q

Reusability Software Configuration Management Manage Test Library Automated Test Data generator Motivation Beta / Internal Beta Operational scenario

Best Practices…

Q

System Tests– Regression Testing– e-Security Testing– Usability Testing– Disaster Recovery Testing– Compatibility testing (multi-platform)– Performance testing

Best Practices…

QRequirements Traceability

Tracing business requirements To business objectives To design To code To test cases

Best Practices…

QDesign – to - Test

Simple Logic Proper flow

Design of Test Cases

Maximum code coverage Defect prone test case

Best Practices…

Q Reviews

Requirements review Design review Code review Test case review

Identifying the Quality levels Identifying Non-Testable requirements Planning the System testing

Benefits

Best Practices…

QProduction data

Environment issues Data Size issues Performance

Best Practices…

QTraceability of defects

Traceability of defects

– Requirements– Code

Decision to release

– Feature– Code– Re-inspect– Re-construct

Best Practices…

QTrain the Testers

Knowledge required in– Functional– Technical

– Testing Train testers

– Testing methodology

Best Practices…

QReusability

Software components Test artifacts and Test cases Generic test case components Automated Test scripts

Benefits

Minimal testing Good Quality Improved productivity

Best Practices…

QSoftware Configuration

Management

Requirement changes affect– Test Plan– Test strategy– Requirements traceability matrix– Test Automation scripts– Test cases– Test effort

Best Practices…

QManage Test Library

Develop and Manage – Test cases– Test scripts– Best practices– Test result report– Test analysis reports

Best Practices…

QAutomated Test Data generator

System testing require more data for execution

Probability of finding fault in a black box testing approach Y= 1-(1-x) N

Where ‘x’ is the probability that the random input will cause the failure

N is the random black box probes.

Best Practices…

Q

Awards Rewards Team building activities

MotivationBest Practices…

QBeta / Internal Beta

Beta for a small group of clients

Internal Beta for a small group of the organization

Product release to the groupUncover DefectsFix Defect

Feedback

Stable Product

Best Practices…

QOperational scenario

Test Case preparation based upon the – Legacy system failures– Previous iteration failures– User focused test – Operational failures– Production support team

Best Practices…

Q

Challenges                          

Q

When to Stop Testing Did I achieved the Quality Goals Selection of Testers Estimation of Test Effort Optimal Test effort Support from the Management Test tool selection

Challenges…

QQuiz

Q

Test Management Metrics

Q Test Managers Job

Activities Planning Monitoring Controlling

ConcernsCostQualitySchedule

QMetrics

Defect Density (SCR/FP) Requirements Stability Shape (release) Trend (Across the releases) Test Group efficiency Planned Vs Executed

Note: ‘Test’ means ‘System testing’ across the Graphs

QDefect Density

0.075630252 0.091743119

0.428571429

00.05

0.10.150.2

0.25

0.30.350.4

0.45

Defects/FP

Release I Release II Release III

Release

Defect Density

Defects/FP

Q Requirements Stability

9

4

0

2

4

6

8

10

No of SCR's

Defect Change Request

Request Type

Requirements Stability

Series1

QType of Defect

1

5

2 2

0

12

3

45

No of Defects

<<None>> Functionality Performance Standards

Defect Type

Defect Type Vs No of Defects (Prod)

No of Defects

12

3

6

0

2

4

6

No of Defects

<<None>> Documentation Functionality Standards

Defect Type

Defect Type Vs No of Defects

No of Defects

QSeverity Defects

344

037

3

17

9

0

10

20

30

No of Defects

I II III

Release

Severity Defects

Critical Major Minor

QTest Group Efficiency

8

27

05

1015202530

No of Defects

UAT Test (QC)Group

Group Vs SCR

Series1

32

3

0

10

20

30

40

NO of Defects

Testing Prod

Environment

Testing Vs Prod

Series1

QTest Planning & Monitoring

Testv Execution Progress

050

100150200250

300350400450500

2000

/10/

01

2000

/10/

08

2000

/10/

15

2000

/10/

22

2000

/10/

29

2000

/11/

05

2000

/11/

12

2000

/11/

19Planned Actual

Planning

• No of BR’s to Test

• Testers Available

• No of Defects acceptable

• Time

QControlling

1. Are we good

Or Was Development Team didn’t do a good job

2. Were we bad before?

3. Why didn’t we found the defects in earlier phases ?

(Apply Systems Engineering)

9

4

0

5

10

No of Defects

Test Prod

Environment

Test Vs Prod

Series1

10

6

0

5

10

No of Defects

Test Prod

Environment

Test Vs Prod

Series1

QIs it Right Time to

Release ?

32

30

10203040

No of Defects

Test Prod

Environment

Test Vs Prod

Series1

Defects per Week during Test

0

5

10

15

20

25

Week

No

Of S

CR

Series1

QWhat if Software is released

Early ?Defects per Week in Test

0123

456

1998/03/05 1998/03/12 1998/03/19

Week

No

of

Def

ects

No of Defects

8

3

0

5

10

No of Defects

Test Prod

Environment

Test Vs Prod

Series1

Defects per Week during Testing

00.5

11.5

22.5

Week

No

of D

efec

ts

No of Defects

3

1

0

1

2

3

No of Defects

Test Prod

Environment

Test Vs Prod

Series1

Q

Exercise

Q

Thank you