step-in summit 2014 · identify a tool for test automation, ex: selenium, qtp, rft, java utility,...

STeP-IN SUMMIT 2014

11th

International Conference on Software Testing

June 2014 at Bangalore, Hyderabad, Pune - INDIA

Practicing the Test Matrix approach to define Test Cases for better coverage

by

Shruti Grover, Advisory Software Engineer, Toshiba Global Commerce Solutions

Copyright: STeP-IN Forum

Published with permission for restricted use in ‘STeP-IN SUMMIT 2014’ in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during ‘STeP-IN SUMMIT 2014’ as per the applicable conditions.

Practices Experience Knowledge Automation

Produced By Hosted By

www.stepinforum.org www.qsitglobal.com

http://www.stepinforum.org/

http://www.qsitglobal.com/

Practicing the Test Matrix approach to define Test Cases for better coverage

ABSTRACT

In software engineering, testing the software for correct behavior before releasing it to end

users is a key activity. The test process can either be done manually or through automation.

Manual testing is the process of directly interacting with the software under test to validate

functionality. Typically, the software test professional plays the role of an end user to ensure

correct operations. To ensure thoroughness of testing, the tester often follows a written test

plan that leads them through a set of important test cases.

Manual testing often becomes laborious considering the multiple combinations that may need to be exercised based on the normal / alternate transactional flow, varying environment configurations, and the virtually infinite input combinations. In absence of providing 100% coverage that would result in thousands or more test case combinations, it becomes non-trivial to determine how much testing is enough to ensure the quality of the software. Creating effective test cases is a skill and that can be achieved by some experience and in-depth study of the application on which test cases are being written. This paper suggests a Test Matrix approach to defining and documenting test cases for providing better coverage and minimizing the test case creation effort. Not only does this mechanism of defining test case ensures full functional coverage, but additionally reduces the effort of defining elaborate test steps and allows for more effective review by peer testers and programmers.

1. CURRENT SCENARIO

The traditional approach of defining test cases is to write manual test cases in: Word / Excel

files, Text Editors, Test Case Authoring Tools -- Quality Center or Rational Manual Tester,

Company in-house tools for test authoring, Tools used for automatic test case generations

based on input set, like-wise.

Few drawbacks of conventional approach is:

Difficult to update & maintain

Laborious, time-consuming, monotonous, prone to human errors – May take hours / days to define elaborate manual test cases, thus posing future maintenance problems!

Limited reusability of test steps & procedures

Difficult to access the overall test coverage – any misses can lead to loss of $$$

Need to invest in licensing of commercial test case authoring tools

STeP-IN SUMMIT 2014Practicing the Test Matrix approach

to define Test Cases for better coverage Paper Publication

1

No easy traceability between manual to automated test cases & no single point maintenance

The Myers View on test coverage & fishing for bugs [The Art of Software Testing, by Glenford

Myers]:

A good test case is one that has a high probability of detecting as-yet undiscovered errors.

It is generally impossible even for trivial examples to test a program using every possible input condition.

In general, the probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section.

Many Test Terminologies – but how to ensure effective test coverage using these aspects:

Equivalence Class Partitioning

Boundary Value Analysis

White-Box Testing

Black-Box Testing

Grey-Box Testing

Independent Inputs

Dependent Inputs

Adhoc Testing

2. INTRODUCTION The basic objective of writing test cases is to validate that the application under test is working as defined through requirements. Defining test cases in plain text one at time becomes monotonous, and makes peer review for completeness difficult and time consuming. Conversion to Automation Scripts becomes challenging, with no easy migration & traceability to manual test cases. Practicing the Test Matrix approach to define test cases not only simplifies and saves time from writing detailed steps, but also allows for effective reviews of scenarios that may be missed otherwise. The basics of these concepts are not new. Many of the required elements such as equivalence class partitioning have been around since the late 1970’s [1]. Also, there have been numerous software test classes [2] that also describe these methodologies. However, combining these together with effective peer review resulting in compact notation and reducing the test case authoring process have resulted in high test efficiency. This allow for a higher proportion of test case execution vs creation – and for higher quality test cases. In addition, it offers easy migration of manual test cases to Automation Suite with single point maintenance.



2

3. TEST MATRIX

a. Initial Assumption

Central to this methodology is that the testers and reviewers are familiar with the

software under test. As such, detailed instructions on how to establish the test

environment, how to create the initial conditions, and how to manipulate the test inputs

are unnecessary. This assumption is essential for the efficiency of this methodology.

b. Defining Test Matrix

A Test Matrix will be smaller in content as compared to the old style of writing elaborate manual test cases. They need to be defined prior to test execution and should be reviewed by all development and test stakeholders. The process of defining a Test Matrix will be broken down into 3 stages: Stage 1 - gather information and determine how to lay out matrix and create left side of matrix – optimal time for developer review Stage 2 - create right side of matrix Stage 3 - execute test cases and complete matrices Now automate them:

Identify a Tool for test automation, ex: Selenium, QTP, RFT, Java Utility, etc

Define a Keyword-driven framework that maps to the Steps defined in Stage 1

While defining Stage 2, identify which can be run for: BVT, FVT-I, FVT-II, Full Suite

Write a Automation Driver Utility that traverses over test matrix keywords and executes test cases based on execution category

c. Creating New Matrices - Stage 1: Create Left Side

As part of defining the left side of the Test Matrix, information about the function under

test should be broken down to smallest level possible. The left side will contain -

1). List options / setup in a separate section at the top of matrix or on a separate page.

2). List expected results information.

3). Make sure matrix can be checked off for completion (or have separate check-off

section).

4). Have a defect information line.

5). Add a line to mark for which test cases need to be Automated, in addition you can

also choose to classify them as: BVT / Smoke, FVT-I, FVT-II, Full Suite.



3

If verifying the User Interface, be sure to include: Usability checklist (if one exists), help,

mouse use, function keys, alternate keys, fast paths.

d. Creating New Matrices - Stage 2: Fill in Right Side

While defining the right side if matrix, ensure that you do not begin filling it until left

side of matrix is stable as requirement specifications / functionality may change.

1). Fill in the options / setup section, actions section and expected results section.

2). Letters and their meanings:

V = valid condition being tested (no error condition should occur)

I (upper case i) = invalid condition being tested (only one invalid condition for a

test case is allowed)

* = don’t care (used in invalid test cases to denote values of other equivalence

classes)

X = can be used to mark options, keys, expected results

1, 2, ...= use numbers of need to show the order of multiple steps to be

performed

Include any additional information that will be helpful to future reviewing of matrices.

3). Mark the Test Case to be Automated & classify them in automation buckets.

e. Creating New Matrices - Stage 3: Execution

For Manual Execution: For the final execution stage, ensure that the squares in matrix are large enough to

write in, in case notes are necessary. Few other things worthy of taking care -

1). Leave extra columns for added test cases while executing.

2). Mark success/failure for each test case.

3). Fill in date last run and defect information.

Any calculations done or information that was discovered during the execution should

be noted on the matrix for future use.

For Automated Execution:

1). Define Automation Keywords for various Test Steps.

2). Automation Keywords implemented in the Tool’s scripting language.

3). Define an Automation Driver Utility that traverses over the test cases marked for

Automation and executes the keywords.



4

Figure 1: A sample text matrix - All Items Independent

In the Test Matrix depicted in Figure 1, observe how the left side and right side information is populated. Left side defines: Initial Conditions, Inputs, and Outputs. Please note that all Inputs in this case are independent of each other. With this context observe how Valid and Invalid test cases have been defined along with their expected results in the right side of the test matrix. Observe how various test steps are identified by a keyword that will be interpreted by an Automation Driver Script to run the same Manual Test Matrix in an automated manner.



5

Figure 2: A sample test matrix - Items 1 & 2 Dependent, Item 3 Independent

In the Test Matrix depicted in Figure 2, please note that all Inputs Items 1 & 2 Dependant on

each other while Item 3 is independent. With this context observe how Valid and Invalid test

cases have been defined along with their expected results on the right side section of the test

matrix. Observe how various test steps are identified by a keyword that will be interpreted by

an Automation Driver Script to run the same Manual Test Matrix in an automated manner.



6

FV Example – Payroll Application Specification

Two Inputs: 1. Employee ID

– 5 digits – less than 50000 indicated temporary employee – 50000 or greater indicates a permanent employee – Partition Inputs::

– 5-digit, temp ID (valid) – 5-digit, permanent ID (valid) – > 5 digits (I) – < 5 digits (I) – Other (I) i.e. non-numeric

2. Number of Hours worked – Pay is $6.00/hr for first 40 hours – Over 40 hours, time and a half – Partition Inputs::

– 0 - 40 (V) – > 40 - MAX (V) – <0 (I) – > MAX (I) – Other (I) i.e. non-numeric

Outputs: – Name – Employee type – Gross Pay



7

Figure 3: A sample test matrix – Payroll Application Specification

Figure 3 depicts a more real-time example where test inputs of: ID number and Hours worked are listed. Both items are independent in this case and on the right side observe how the combination of valid / invalid inputs result into different outputs. Again observe how various test steps are identified by a keyword that will be interpreted by an Automation Driver Script to run the same Manual Test Matrix in an automated manner.



8

4. TEST AUTOMATION LEVERAGING THE TEST MATRIX Keywords defined in Code

Implementing the Keyword

Integration with Test NG for category-wise execution



9

5. TEST TECHNIQUES AND BEST PRACTICES

a. Testing Techniques to use for Test Matrix

Ensure the usage of Equivalence Class Partitioning (ECP), Boundary Value Analysis (BVA), and Experience based testing while defining test cases using the Test Matrix approach. A brief understanding on each of them is mentioned.

ECP is technique where the input space is divided into partitions so that items in each partition are treated the same. Both valid and invalid classes are considered, but at least one element from each partition should be picked to test.

BVA is a technique to test the boundaries (including machine boundaries) and to test close to the boundaries.

When a specific number of items is to be input, try: too many, too few, zero. For an ordered set, check first and last elements e.g. {1, 3, 5, 7, 9}.

For output: test maximum output, attempt to exceed the maximum, test minimum output, test no output.

Intuition and Experience based testing is a third technique that can be incorporated to write high quality test matrices. Using this technique, tester should anticipates or guess problem areas and look for vague or incomplete specifications or documentation. This is highly dependent on individual skills and experience. This technique must be used with other testing approaches; as it is not effective when it is the only test selection process. This testing is also named as:

Error guessing

Artistic testing

Ad hoc testing

“What if” testing

b. Best Practices for Test Matrix

For authoring Test Matrices ensure the following best practices: 1). For all matrices, include:

Initial Conditions - Indicate what the situation must be before test case starts

Inputs/Parameters - using V, I, and *, indicate what parameters to use

Verification - using prose or tables, indicate HOW results will be verified

Results - using X, indicate WHAT results are expected 2). Values that span a range must have:

Min

Max



10

min-1

max+1

0 and/or -1 if they aren’t already included 3). For test cases that generate output files, add test cases that generate I/O error (ex. attempt to save to full disk) - add comment to verify output file. 4). Strings should be tested for:

blank string, blank within string

string too long, min/max length char string

invalid characters/value

0 char string (null string) 5). NLS considerations

Accent characters

Code pages

Currency symbols

Decimal separators

Time format (12h vs 24h)

Date format 6). GUI Best Practices For each panel:

Ensure each field/button/item can be accessed by using - forward/backward tab, cursor keys

Test all PF keys; select Ctrl, Shift, Alt key

For those panels that allow options, select all options and no options

For those panels that have text or comments, test the text or comment as though it were part of the specification

For each data entry field, test: no data; too much data; invalid syntax; Help function; everything help says can/should not be done;

Include the following in the notes section of the matrix: Execute this test once using keyboard and once using mouse

6. BENEFITS

Grid format gives visual intuitiveness in cross-checking if all necessary combinations are

exercised

In absence of providing 100% coverage that would result in thousands or more test case combinations, it helps determine how much testing is enough to ensure the quality of software



11

Reduces the effort of defining elaborate test steps – faster turnaround in authoring test cases (from several hours / days the effort reduced to – 1 or 2 days per functional area!!)

Allows for more effective review by peer testers and programmers

Reuse of same test matrix for driving automation test cases based on different test scope

Reuse of same test matrix for a keyword driven automated test execution approach

Flexibility to adapt the usage of this approach with any Test Automation Tool – with a simple Excel Driver & Automation Keyword implementation

Provides significant value for time & effectiveness – which is essential for Agile Iteration based SDLC model

7. CONCLUSION With the Test Matrix approach of test case authoring there is a clear time and effort saving, as the task of defining multiple elaborate test cases and maintaining them is now transformed to a single matrix per functionality with a single document update. The test definition for a feature that would need many test cases to thoroughly ensure its quality, need not now be defined with detailed steps. Instead a smarter approach is to define the test steps in form of Input / Output in Test Matrix. This test matrix will also ensure that combinations of environments, valid / invalid entries, and exceptional flows are thoroughly covered. Being written in a grid format gives a visual intuitiveness in cross-checking that all necessary combinations are being exercised. This allows for a more efficient review by peer testers and programmers whose time is already taxed in a typical software development project. The Test Matrix approach has value for time and effectiveness – which is essential to achieve faster and shorter software development cycles. The savings from test case planning and authoring can now be reallocated to smarter and more complete test execution as we create one document that serves the purpose of all combinations.

8. REFERENCES

[1]. Myers, Glenford Myers. The Art of Software Testing [2]. Cheatham, Mickey, et. al., Software Test:Unit Through System and Acceptance Test



12

step-in summit 2014 · identify a tool for test automation, ex: selenium, qtp, rft, java utility,...

Documents