qa documentation pp
TRANSCRIPT
Software and System
Test Documentation
IEEE 829 - Standard for Software Test Documentation
TEST PLANNING AND CONTROL
Master Test Plan
Level Test Plan
TEST ANALYSIS AND DESIGNLevel Test Design
Level Test Case
TEST IMPLEMENTATION AND EXECUTION Bug Report
Level Test Procedure
EVALUATING EXIT CRITERIA AND REPORTING
Interim Test Status Report
Test Log
Level Test Report
TEST CLOSURE ACTIVITIES Master Test Report
LEVELS OF TEST PLANDevelop Master Test Plan
Develop Detailed Test Plans
Unit Test Plan
Integration Test PlanSystemTest Plan
AcceptanceTest Plan
Level of Test Plan defines what the test plan is being created for.
Test Plan document follows the same structure for each level of test plan. The only difference is the content and detail.
Unit+Integration Test Plans Example
System+Acceptance Test Plans Example
Master Test Plan Example
TEST PLANNING AND CONTROL
MASTER TEST PLAN
The purpose of the MTP is to:
provide the overall framework for all the testing activitiesdefine the scope of the testing
identify whether there is any unnecessary duplication of testing taking placeidentify the departures from the Test Process documentation set
define the approach to each stage of testingspecify the ABC project’s staff’s responsibilities for testing activities for each stage of testing
A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process.
A test plan that typically addresses multiple test levels.
A test plan that typically addresses one test phase.
Test Plan
The format and content of a software test plan vary depending on the processes, standards, and test management tools being implemented. Following format provides a summary of what a test plan can/should contain.
1) Test Plan ID: Unique No or Name2) Introduction: Provide an overview of the test plan, specify the
goals/objectives.3) Test Items: Modules/ Functions/ Services/ Features/ etc.4) Features to be tested: Responsible Modules for Test design5) Features not to be tested: Which feature is not to be tested
and Why? 6) Approach: List of selected testing techniques to be applied on
above specified modules in reference to the TRM (Test Responsible Matrix)
7) Feature pass or fail criteria: When a feature is pass or fail description
8) Suspension criteria9) Test Environment: Required software & Hardware to be tested
on above features10)Test Deliverables: Required testing document to be prepared11)Testing Task: Necessary tasks to do before start every feature
testing12)Staff & Training: Names of selected Test Engineers & training
requirements to them13)Responsibilities: Work allocation to every member in the team14)Schedule: Dates & Times of testing modules15)List & Mitigation: Possible testing level risks & solution to
overcome them16)Approvals: Signatures of Test plan authors & Project Manager /
QA
Wha
t to
test
?H
ow to
test
?W
hen
to
test
?
TEST DESIGN LEVELS STRUCTURE BASED ON THE V-MODELOverall Business Requirements
Acceptance Test Design
Software requirements
System Test Design
High level requirements
Integration Test Design
Low level requirements
Component Test Design
Coding Unit Test Design
Unit Test Execution
Component Test Execution
Integration Test Execution
System Test Execution
Acceptance Test Execution
TEST ANALYSIS AND DESIGN
TEST DESIGNTest Design Phase – In software engineering, test design
phase is a process of reviewing and analyzing test basis, selecting test design techniques and creating designed test cases, checklists and scenarios for testing software.
Test Design Specification - it is a document that describes features to be tested and specifies list of all test scenarios or test cases, which should be designed for providing the testing of software. Basically test design is the act of creating and writing test suites for testing a software.
Test design could require all or one of:1) Knowledge of the software, and the business area it operates on2) Knowledge of the functionality being tested 3) Knowledge of testing techniques4) Planning skills to schedule in which order the test cases should be designed, given the effort, time and cost needed or the consequences for the most important and/or risky features
Review and Analyze Test
Basis
Select Test Design
Techniques
Create Test Design
Specification
Create Test Cases
Specification
Test Plan
Requirements Mock-
ups
Test Design
Specification
Test Case Specificati
on
BASIC TEST DESIGN STEPS:
• The internal factors that influence the decision about which technique to use are: – Tester knowledge and experience – Expected defects – Test objectives – Documentation – Life cycle model
• The external factors are: – Risks – Customer requirements – System type– Time and budget
Choosing A Test Design Technique
According to IEEE-829 standard template structure looks in the following way:
1. Test Design Specification Identifier1.1 Purpose1.2 References1.3 Definitions, acronyms and abbreviations
2. Features to be Tested3. Approach Refinements4. Test Identification
4.1 <Test Item 1>4.2 <Test Item …>4.3 <Test Item N>
5. Feature Pass/Fail Criteria
Test Design Specification Structure
Belo
w –
th
e
exp
lan
ati
on
3)Approach Refinements section describes the following:– Specific test techniques to be used
for testing features or combinations of features
– Types of testing which will be provided
– Methods of analyzing test results– Test results reporting– Whether automation of test cases
will be provided or not– Any other information which
describes approach to testing
Test Design Specification Structure
1) Test Design Specification Identifier section covers:– Purpose of the document– Scope of the document– List of references which should
include references on test plan, functional specification, test case specification, etc.
– Definitions, acronyms and abbreviations used in Test Design Specification
2) Features to be Tested identifies test items and describes features and combinations of features that are the object of this design specification. Reference on Functional Specification for each feature or combination of features should be included.4) Feature Pass/Fail Criteria specifies the criteria to be used to determine whether the feature or feature combination has passed or failed.
The following items can be considered as “pass / fail criteria”:
1) Feature works according to stated requirements
2) Feature works correctly on the test platforms
3) Feature works correctly with other modules of application
4) All issues with High and Medium Priority will be verified and closed
5) Test Identification section is separated to sub-section
according to the amount of test items identifying future
documentation which will be created for testing features or
combinations of features that are the object of this design
specification.
Features can be covered by test objectives in different ways
depending on projects needs, approaches for testing etc.
Three examples of such
coverage:
Feature covered by test cases
Feature covered by test
scenarios Feature covered
by checklists
Test Design With Scenarios
Test Design With Test Cases
Test Case Specification
Test Specification – It is a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Revision History - This section contain information like Who created the
test specification? When was it created? When was the last time it was updated?Feature Description – A brief description of what area is being tested.What is tested? – An overview of what scenarios are tested.What is not tested? - Are there any areas that are not being tested. Nightly Test Cases – A list of the test cases and high-level description of what is tested whenever a new build becomes available.Breakout of Major Test Areas - It is the most interesting part of the test specification where testers arrange test cases according to what they are testing.Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.Security tests – Any tests that are related to security.Accessibility Tests – Any tests that are related to accessibility.Performance Tests - This section includes verifying any performance requirements for your feature.Localization / Globalization - tests to ensure you’re meeting your product’s Local and International requirements.NOTE: Test Specification document should prioritize the test case easily like nightly test cases, weekly test cases and full test pass etc:Nightly - Must run whenever a new build is available.Weekly - Other major functionality tests run once every three or four builds.Lower priority - Run once every major coding milestone.
Con
ten
ts o
f a T
est
Sp
ecifi
cati
on
:
Test cases are a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. There must be at least one test case for each requirement for traceability.
Test Cases
Test Case
Test Step – specifies an
action to perform, and the expected response of the test application.
For example: Action : Type the password in the password box,
Expected result: Your password
should be dotted / hidden.
Test Case – a list of test steps. Also
defines the environmental
situation and may link to related
bugs, requirements etc.
Test Scenario – usually comes directly from
business requirements or
user story. Scenario contains a list of test cases
and often their sequence.
Test scenario Test case Test Step
Standard fields of sample test case template:
Test case ID: Unique ID for each test case. Test priority (Low/Medium/High): This is useful while test execution. Test priority for business rules and functional test cases can be medium or higher whereas minor user interface cases can be low priority. Module Name – Mention name of main module or sub module.Test Designed By: Name of testerTest Designed Date: Date when wroteTest Executed By: Name of tester who executed this test. To be filled after test execution.Test Execution Date: Date when test executed.Test Title/Name: Test case title.Test Summary/Description: Describe test objective in brief.Pre-condition: Any prerequisite that must be fulfilled before execution of this test case. List all pre-conditions in order to successfully execute this test case.Dependencies: Mention any dependencies on other test cases or test requirement.Test Steps: List all test execution steps in detail. Write test steps in the order in which these should be executed. Make sure to provide as much details as you can. Test Data: Use of test data as an input for this test case. You can provide different data sets with exact values to be used as an input.Expected Result: What should be the system output after test execution? Status (Pass/Fail): If actual result is not as per the expected result mark this test as failed. Otherwise passed.Notes/Comments/Questions: To support above fields if there are some special conditions which can’t be described in any of the above fields or there are questions related to expected or actual results mention those here.
Fields of sample test case template, if necessary:
Defect ID/Link: If test status is fail, then include the link to defect log or mention the
defect number.
Test Type/Keywords: This field can be used to classify tests based on test types. E.g. functional, usability, business rules etc.
Requirements: Requirements for which this test case is being written. Preferably the exact
section number of the requirement doc.
Attachments/References: This field is useful for complex test scenarios.
Automation? (Yes/No): Whether this test case is automated or not. Useful to track automation
status when test cases are automated.
The purpose of an LTPr is to specify the steps for executing a set of test cases or, more generally, the steps used to exercise a software product or software-based system item in order to evaluate a set of features.
1. Identification (Each Test Procedure Specification should be assigned a unique identifier.)
2. Purpose (explain the purpose of the test procedure and reference any test cases it executes.)
3. Special Requirements (list any special hardware, software or training requirements for this procedure.)
4. Procedure Steps (actual steps of the procedure are described. IEEE has listed several steps:
Log: explain methods / formats for logging results of test execution.
Set up : actions necessary to prepare for the execution of procedure.
Start : actions necessary to begin execution of the procedure.
Proceed : actions necessary during the execution of the procedure.
Measure : explain ho test measurements need to be made.Shutdown : actions necessary to suspend testing when
some unscheduled event happens.Restart : actions necessary to restart the procedure. Stop : actions necessary to bring execution to systematic
halt.
Level Test Procedure
Test
Pro
ced
ure
S
pecifi
cati
on
sh
ou
ld h
ave t
he fo
llow
ing
ele
men
ts:
Test Procedure
TEST IMPLEMENTATION AND EXECUTION
ID Unique identifier given to the defect.
Project Project name.
Release Version Release version of the product. (e.g. 1.2.3)
Module Specific module of the product where the defect was detected.
Detected Build Version
Build version of the product where the defect was detected (e.g. 1.2.3.5)
Summary Summary of the defect. Keep this clear and concise.
Description Detailed description of the defect. Describe as much as possible but without repeating anything or using complex words. Keep it simple but comprehensive.
Steps to Reproduce Step by step description of the way to reproduce the defect. Number the steps.
Actual Result The actual result you received when you followed the steps.
Expected Results The expected results.
Attachments Attach any additional information like screenshots and logs.
Remarks Any additional comments on the defect.
Defect Severity The seriousness of defect with respect to functionalityHigh(Show Stopper):-Without fixing this defect tester not able to
continue testing.Medium:-Able to continue but mandatory to fix.Low:-Able to continue testing but may or may not to fix.
Defect Priority The importance of defect fixing with respect to customer interest.
Reported By The name of the person who reported the defect.
Assigned To The name of the person that is assigned to analyze/fix the defect.
Status The status of the defect. (new/reopen)
Fixed Build Version Build version of the product where the defect was fixed (e.g. 1.2.3.9)
DEFECT REPORT
Test Log
Test Log is details of what tests cases were run, who ran the tests, in what order they were run, and whether or not individual tests were passed or failed. The purpose of the Level Test Log is to provide a chronological record of relevant details about the execution of tests. An automated tool may capture all or part of this information.
Test Log
EVALUATING EXIT CRITERIA AND REPORTING
Interim Test
Status Report
The purpose of the ITSR is to summarize
the results of the designated testing
activities and optionally to provide
evaluations and recommendations
based on these results.Interim Test Status Report
Eight Interim
Reports:
Functional Testing Status
Functions Working Timeline
Expected verses Actual
Defects Uncovered Timeline
Defects Uncovered verses
Corrected Gap
Timeline
Average Age of
Uncorrected
Defects by Type
Defect Distributio
n
Relative Defect
Distribution
Testing Action
Functional Testing Status Report
Fully tested
Tested with open defects
Not tested
This report will show percentages of the
functions which have been:
Functions Working Timeline
This report will show the actual plan to have all functions working
versus the current
status of functions working.
An ideal format could be a line
graph.
Expected versus Actual Defects Uncovered Timeline
This report will provide an analysis between the number of defects being generated against the expected number of defects expected from the planning stage.
This report, ideally in a line graph format, will show the number of defects uncovered versus the
number of defects being corrected and accepted by the testing group.
If the gap grows too large, the project may not be ready when
originally planned.
Defects Detected versus Corrected Gap
This report will show the average days of outstanding defects by type. In the planning stage, it is beneficial to determine the acceptable open days by defect type.
Average Age Uncorrected Defects by Type
This report will show the defect distribution by function or module. It can also show items such as numbers of tests completed.
Defect Distribution
This report will take the previous report (Defect Distribution) and normalize the level of defects. An example would be one application might be more in depth than
another, and would probably have a higher level of defects. However, when normalized over the number of functions or lines of code, would show a more
accurate level of defects.
Relative Defect Distribution
This report can show many different things, including possible shortfalls in testing. Examples of data to show might be
number of Sev 1 defects, tests that are behind schedule, and other information that would present an accurate testing
picture.
Testing Action
Level Test
Report (LTR)
The purpose of the Level Test Report
(LTR) is to summarize the results of the
designated testing activities and to
provide evaluations and
recommendations based on these
results
Master Test
Report
The purpose of the MTR is to summarize the results of the
levels of the designated testing activities and to
provide evaluations based on these results. This report may be used by any organization using the MTP. Whenever an
MTP is generated and implemented, there needs to be a corresponding MTR that describes the results of the
MTP implementation
Master Test Report
TEST CLOSURE ACTIVITIES