Practical Testing DefinitionsFor Testing Mobile Devices

Purpose

• This document defines general tester roles and different test levels on a high level which are applicable for any field

• These test levels are then broken down to useful practical definitions for when testing mobile devices

Definition Overview

Unit Test

Integration Test

System Test

IOT

Software Engineer

Software Engineer in Test

Test Engineer

ValidationBlack Box

Gray Box

White Box

Test Level

Test Role

Level Overview

Unit

Unit Test

Unit Unit UnitUnit Unit Unit Unit Unit

System

External

Integration Test

System Test

Validation

IOT

Theoretical Definition

Validation Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]

Verification Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. [ISO 9000]

Interoperability The capability of the software product to interact with one or more specified components or systems. [ISO 9126]

System Test System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic.[IEEE]

Integration Test Integration testing works to expose defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a system. [Beizer, Boris]

Unit Test Testing of individual hardware or software units or groups of related units [IEEE]

Practical Definition: Unit Test

• In computer programming, unit testing is a method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine if they are fit for use. Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function or procedure. In object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method. [2]

• API testing is done using a White Box approach– Functionality– Reliability– Efficiency

• Main Challenges of API Testing can be divided into following categories– Parameter Selection – Parameter combination – Call sequencing – Different system states for the above

• Unit test techniques such as Boundary Value Analysis, State Transition Testing, Equivalence Partition Testing, Cause-Effect graphing, etc.

Practical Definition: Integration Test

• Testing all ISO 9126 software quality attributes [3] during feature development and when a feature or application is complete and all parts are integrated

– All the different parts have been unit tested but are now integration tested as soon as part of the feature is testable either through API testing or manual or automatic testing through the UI

– When the feature or application is complete additional testing is done to secure any system impact caused by the feature or application

• API testing is done using a Gray Box approach– Functionality– Reliability– Efficiency

• Focus on system impact of changes that are being integrated, based on system dependencies, securing that there are not regressions to other parts of the system

• A feature could be a new integrated phonebook application, a new camera implementation, a new Bluetooth profile, a new WiFi feature, changes in the file system, or something similar

Practical Definition: System Test

• User Scenarios [1]

• System Capabilities– System Power Consumption– System Performance– System Stability– System Memory Usage– System Stress

• User Experience / Usability

• Customer Customizations– Testing the mobile device with different customer customizations

• Application Composition Testing– Testing a complete application suite that is going to be released with the mobile device

• System Upgrade Testing– Securing that it is possible to upgrade the system without any unwanted side effects

• System Age Testing– Securing that the mobile device works as expected after extended periods of usage

• Localization Testing– Translation and language testing

Practical Definition: IOT

• Device IOT – Testing the mobile device with different third-party appliances– TVs– Laptops– Cameras– Speakers– Etc.

• Accessory IOT – Testing the mobile device with different mobile device accessories, such as headphones

• SIM IOT – Testing the mobile device with different SIM cards

• Service IOT– Testing the mobile device with different services

• External Storage IOT – Testing the mobile device with different types of external storage devices

• Network Dependant Testing– Testing the mobile device in different operator networks

Practical Definition: Validation

Analysis of Verification

Results

Verification Results

Product Risks

Risk Management Model

Validation Test Activity

Customer Requirements

Regulatory Requirements

Test Gap Report

Validation Guidelines

Validation Report

Analysis of Validation

Results

Quality Criteria

Quality Report

IOT, System, or Integration Tests

Company Internal179/038 13-LXE 110 0048 Uen PA4 Interview with a Tester2012-02-1511

Not in Scope

• Time aspects have not been discussed in this document – the test levels defined can be used at any point in the project, even though it may be difficult to test the system until a certain maturity has been reached

• Automation has not been discussed – all test levels can be automated or executed manually, even though unit tests are preferably automated, and validation is often run manually due to the nature of the activity

• Scripted or exploratory testing has not been discussed – both approaches can be used for any manual testing

• Regression testing can be done on any testing level at any stage during a project whenever there are possible regressions of the system

Summary

• This is just one way to define these different test levels and roles

• Using these definitions to clarify test activities and test roles will facilitate communication and support testers in understanding their responsibility

Reference

[1] An Introduction to Scenario Testinghttp://www.kaner.com/pdfs/ScenarioIntroVer4.pdf[2] Wikipedia – Unit Testinghttp://en.wikipedia.org/wiki/Unit_testing[3] ISO 9126http://en.wikipedia.org/wiki/ISO_9126