introducing the new software testing standard

T24 Concurrent Class

10/3/2013 3:00:00 PM

"Introducing the New Software

Testing Standard"

Presented by:

Jon Hagar

Consultant

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Jon Hagar

Grand Software Testing

Jon Hagar is a systems-software engineer and tester consultant supporting software product

integrity and verification and validation, with a specialization in embedded and mobile software

systems. For more than thirty years Jon has worked in software engineering, particularly testing,

supporting projects including control system (avionics and auto), spacecraft, mobile-smart

devices, IT, and attack testing of smart phones.

9/19/2013

1

1

Copyright 2013 Jon Hagar Grand Software Testing GST

Special Topics T24

Introducing the New

Software Testing Standard:

ISO/IEEE/IEC 29119

Jon Hagar

[email protected]

Grand Software Testing (GST), LLC

303-903-5536


Scope

� Purpose of standards

� Types of standards

� Development of standards

� Overview of ISO29119

� Applicability

� Recent developments

� Timeline

� Future work

9/19/2013

2


Some Preliminaries � Under disclosure, I am the IEEE Project Editor for

ISO/IEEE/IEC 29119 Software Testing Standard

� Before you decide to roast me . . .please recognize � Standards are a fact of life for many industry professionals not just

testers

� Standards form part of the business for many companies , customers, and countries

� Many of us end up in a mixed relationship with standards

� Standards can be a basis for research and change

� Industry has not had a comprehensive software testing standard


What are standards?

� Guideline documents—not compulsory unless

mandated

� Agreements because they often reflect a certain level of

consensus within a community

“Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationallyrecognized industrial, professional, trade associations or governmental bodies”

- ISO

9/19/2013

3


How can a standard “help?”

� Standards are a basis for:� Communication – common terminology

� Professionalism and criticism

� Conformance/compliance schemes

� Benchmark of ‘industry practice’ (not best, may not be good for all, etc.)

� Contracts and regulations

� Interoperability and consistency. . .

Standards describe a current ‘body of knowledge’ that provides the basis for a professional discipline


ISO29119 Standard - My Viewpoint

� Any software standard is based on limited history (rear

looking) and developed based on consensus of many voices

� Yet still may represent only some of viewpoints of industry

� But does not represent all states of the art

� Standards need to be inclusive and ethical

� As with many “schools,” views and opinions but with NO

“best”

� Not claimed to be perfect

� Most projects/professionals will use “tailoring”

9/19/2013

4


Standardization Bodies


Motivation for ISO29119

� Conflicts in definitions, processes and procedures� “One ring to rule them all” — standards to be replaced by one

� e.g., IEEE 829, IEEE 1008, BS7925-1/-2, IEEE 1028

� Users do not know which standard to follow

� Lacking in current standards or incomplete:� Organizational areas

� e.g., Test Policy and Organizational Test Strategy

� Project Test Management

� BS7925 only covers unit testing

� General processes

� Common functional techniques missing

� Coverage of non-functional testing

9/19/2013

5


ISO/IEC29119 –Structure and Flow

BS7925-1

BS7925-2 IEEE 829

Concepts & Vocabulary

Part 1

Process

Assessment

Testing

Techniques

Part 4

Documentation

Part 3Part 2

Processes

Keyword-Driven

Testing

Part 5 ISO/IEC 33063

ISO 12207ISO 15288Directives

IEEE 1008


A Brief Diversion– Schools of Testing� Initially proposed by Pettichord (workshop notes) et. al.

� Labels used here with some liberty (to be positive)

� Context driven is one, but there are others, possibly� Agile (the rage for the last 10 years)

� Academic (researchers)

� Quality assurance (whatever that is)

� Process based

� “None” (not an official name, but I see these testers all the time)

� Some “schools” may not recognize themselves (by the names)� Schools are common in art, music, sciences, and other professions

� But many “artists” cross the school lines of art

� ISO29119 aims to be “school independent”, but will have a process centric view but with liberal ability to tailor

9/19/2013

6


Do Testers Need Standards? –

Yes, maybe, but

� Standards support common communication within the topic

� Common reference points

� Starting point for research, usage (pro & con), critic

� Maturity is an issue but a baseline serves as sounding board

and common reference point for “Scientific” methods

� An international benchmark

� Thinkers and researchers can prove/disprove benchmark(s)

� Part of being in a profession (but only part)

� Standards have dangers


My View – Potential Dangers

� Not every book, school, paper, ideal will be represented

� There must be a vetting, upgrade, and maintenance effort

� Will not make everyone happy, so in most cases must be tailored by anyone using them

� Will be subject to “misuse” by various groups� Unthinking audits/reviews

� May stop some people from thinking for themselves (but many do that anyway)

� Will (always) be imperfect

9/19/2013

7


Part 1: Concepts & Vocabulary

SOFTWARE TESTING CONCEPTS

Scope, Conformance, Normative References

TESTING IN DIFFERENT LIFE CYCLE MODELS

ROLES AND RESPONSIBILITIES IN TESTING

ANNEXES – Metrics, Examples, Bibliography

DEFINITIONS

Te

st:

Ap

pro

ach

, B

asis

, M

eth

od

s

-R

isk

Ba

se

d T

esti

ng


Part 2: Testing Processes

TEST MANAGEMENT PROCESSES

ORGANIZATIONAL TEST PROCESS

DYNAMIC TEST PROCESSES

9/19/2013

8





Instantiating Testing Processes

Ref: S. Reid


Organizational Test Specification

Develop Organizational

Test Specification

(OT1)

Monitor and Control Use of Organizational

Test Specification

(OT2)

UpdatedOrganizational Test Specification

UpdateOrganizational

Test Specification

(OT3)

Controlled Organizational Test Specification

Organizational Test Processes

9/19/2013

9




TEST

PLANNING

TEST

MONITORING &

CONTROL

TEST

COMPLETION

ORGANIZATIONAL TEST

DOCUMENTATION

FEEDBACK ONORGANIZATIONAL TEST DOCUMENTATION

TEST PLAN UPDATES

TESTPLAN

TESTCOMPLETIONREPORT

DYNAMIC TEST

PROCESSES

TEST

MANAGEMENT

PROCESSES

TEST PLAN,TEST COMPLETION REPORT,TEST MEASURES

TEST MEASURES

TEST PLAN,CONTROL DIRECTIVES

TEST PLAN,CONTROL DIRECTIVES

Test Management Processes


OrganizeTest Plan

Development

Identify & Estimate Risks

Design Test Strategy

Determine Staffing and Scheduling

Document Test Plan

Schedule, Staffing Profile

Test Strategy

Analyzed Risks

Scope

Identify Risk Treatment

Approaches

Gain Consensus on

Test Plan

ApprovedTest Plan

DraftTest Plan

Test Plan Publish

Test Plan

Understand Context

TreatmentApproaches

Test Planning Processes

9/19/2013

10


Standard Primarily has a Risk-based Process Flavor

� Risk based testing is a primarily “good” practice

� Other bases/approaches (Agile, exploratory, model-based, math-

based, etc.) are included

� All parts may need “tailoring” (few efforts would be “full” conformance)

� Use of the standard implies skills, thinking, dependence on

context (“it depends”), etc.


Test Monitoring & Control Process

Monitor

Report

measures

ControlSet-Up

[testing complete]

[testing incomplete]

control directives

testprogress

info

test progressinfo

test plan

testmeasures

test status report

testcontrolinfo

...TEST PROCESSES...

Dynamic/Management

9/19/2013

11



TESTSPECIFICATION

[NO ISSUESNOTICED]

TEST ENVIRONMENT READINESS REPORT

TESTENVIRONMENT

REQUIREMENTS

INCIDENTREPORT

TEST DESIGN &

IMPLEMENTATION

TEST

ENVIRONMENT

SET-UP

TEST

EXECUTION

TEST

INCIDENT

REPORTING

(LEVEL) TEST MANAGEMENT PROCESS

[ISSUE NOTICED orRETEST RESULT]

TEST RESULTS

CONTROL DIRECTIVES TEST

MEASURES

(LEVEL)TEST PLAN

Dynamic Test Processes


Part 3 – Test Documentation

TEST DOCUMENTATION

ANNEXES - EXAMPLES

Scope, Conformance,

Normative References

Se

lect

a s

ub

se

t o

f d

ocs

9/19/2013

12


Part 3: Test Documentation� Organizational test documentation

� Test policy

� Test strategy

� Project test documentation

� Project test plan

� Test project completion report

� Test Level documentation

� Test plan

� Test specification

� Test results

� Anomaly reports

� Level test status report

� Test environment report

� Test level completion report

� Appendices

� Examples of documents at each level of testing


Part 4 – Test Techniques

TEST COVERAGE MEASUREMENT

Scope, Conformance, Normative References

ANNEXE – TESTING OF QUALITY CHARACTERISTICS

ANNEXE – SELECTION OF TECHNIQUES

ANNEXE – TEST TECHNIQUE EFFECTIVENESS

TEST DESIGN TECHNIQUES

Functional Structural

9/19/2013

13


Part 5- Keyword-Driven Testing

� Part 5 will address:

� Concept

� Applicability

� Interfaces

� Approach

� Part 5 WD was sent out in May and next draft due in Nov 2013


Applicability of ISO/IEEE 29119� It is intended to be a generic standard to support all

domains

� Some domains are more standards oriented such as:

� safety-related

� telecoms

� Some domains will become more regulated such as:

� financial – banks, stock markets, etc.

� Needs trails in all domains and project types

� Applicability to sequential, iterative and agile

� Applicability to new as well as maintenance projects

� Applicability to IT and embedded

9/19/2013

14


Discussion Point: Thoughts?

� Do you think we need standards?

� If we use a standard does this mean we do not think?

� What should the test community do?


My Thoughts� Thinking testers with a context-driven/based viewpoint

� Use standards when a customer or project requires them e.g., in regulated environments

� Understand where standards are “good” (worthwhile) and bad (not worthwhile)

� Use standards as a professional reference work, but know when and how to break the “rules” via tailoring

� How should the test community react?

� Engage to identify and address both the good and bad

� Do not use standards in all situations (test projects)

� Use materials for constructive criticism

� They will have many concepts which need to be “over turned”

� Drive the scientific review and healthy debated

9/19/2013

15


Timelines – Current Status

May 2010

May 2011

May 2012

May 2013

…

Working Draft (WD)Committee Draft (CD)Draft International Standard (DIS)Final Draft International Standard (FDIS)Final International Standard (FIS)

Parts 1, 2, 3Parts 1, 2, 3

Part 4Part 4

WDWDCD1CD1

DISDISFDISFDIS

FISFIS

WDWDCD1CD1

DISDISFDISFDIS

FISFIS

CD2CD2CD3CD3

CD2CD2CD3CD3

4

Part 5Part 5

WDWD


For more information

� [email protected]

� For more information and details

� http://softwaretestingstandard.org/

� WG26 website

� http://www.jtc1-sc7.org/

� Access to official documents released by WG 26

9/19/2013

16


Summary

� First attempt at a comprehensive software test

standard is being released

� Even in the presence of standards, a thinking

tester is always needed

�What will you do?


Stuart Reid

Testing Solutions Group117 Houndsditch

London EC3UK

Tel: 0207 469 1500Fax: 0207 623 8459

www.testing-solutions.com

Major Reused Reference Point:

introducing the new software testing standard

Technology