copyright © fraunhofer iese 1999 6 pbr concepts 6 pbr concepts 7 object-oriented inspections 7...

Copyright © Fraunhofer IESE 1999

6 PBR Concepts6 PBR Concepts

7 Object-oriented7 Object-orientedInspectionsInspections

ContentsContentsContentsContents

Slide 1

8 Summary8 Summary

1 Goals1 Goals

3 Why perform3 Why perform inspections?inspections?

2 What is an 2 What is an inspection? inspection?

4 How to evaluate4 How to evaluate inspections?inspections?

5 Reading: a key 5 Reading: a key technical technical dimensiondimension

Oliver LaitenbergerOliver LaitenbergerFraunhofer Institute for Fraunhofer Institute for Experimental Software EngineeringExperimental Software Engineering

Review and Inspection Champions ForumReview and Inspection Champions Forum

Current Trends in Inspection Current Trends in Inspection Technology Technology





Slide 2

8 Summary8 Summary

1 Goals1 Goals





Rostock

BerlinPotsdam

Magdeburg

Dresden

ChemnitzJena

HamburgBremen

Hannover

Dortmund

Oberhausen

Aachen

DarmstadtWürzburg

Erlangen

NurembergKarlsruhe

Stuttgart Freising

Munich

Garmisch-Partenkirchen

Freiburg

Saarbrücken

Braunschweig

HalleCottbus

Kaiserslautern

Fraunhofer IESE: Applied Research and Technology Transfer

Customers in Telecommunication Area Bosch Telecom Deutsche Telecom Ericsson Lucent Siemens AG ...

Competence Areas

Systematic Software Development

Quality and Process Engineering

Product Line Development

Systematic Learning and Improvement

Continuous Education and Training

Transfer Partners Q-Labs Fraunhofer Center Maryland

ISERN ...

E-mail: [email protected]: http://www.iese.fhg.de





Slide 3

8 Summary8 Summary

1 Goals1 Goals





• Goals

• What is an inspection?

• Why perform inspections?

• How to evaluate inspections?

• Reading: A key technical dimension of inspections

• Introduction to PBR concepts

• Object-oriented Inspection

• Summary

Contents





Slide 4

8 Summary8 Summary

1 Goals1 Goals





• provide an overview of current trends in

inspection technology.

• provide some insight into how to evaluate

inspections in practice.

• Introduce the use of reading techniques to

improve inspection effectiveness.

• Providing an understanding of the ideas

behind Perspective-based Reading (PBR),

a particular set of reading techniques.

• Point out some issues related to the

inspection of object-oriented development

artifacts and how they are addressed with

PBR.

Goals of this Talk





Slide 5

8 Summary8 Summary

1 Goals1 Goals





What is an Inspection? Definition Inspection is a static analysis method for

verifying the quality properties of software products.

Characteristics: • Structured, well-defined process.

• Inspection team usually consists of qualified personnel.

• Participants have well-defined roles.

• Reading techniques are used for the defect detection activity.

• Inspection results are documented.





Slide 6

8 Summary8 Summary

1 Goals1 Goals





How to conduct an Inspection?

Planning

Detection

DefectReportForm

Collection

Correction

SoftwareProduct

PlanningForm

Defect Correction

Form

1

2

3

4

organizer

inspector

moderatorinspectorsauthor

author

CorrectedSoftwareProductSoftware

Inspection

DefectCollection

Form

Roles

Activities

Products

Reading Techniques





Slide 7

8 Summary8 Summary

1 Goals1 Goals





“Inspection” - An Overview

Ad-hocChecklistsRSA

Defect-based ReadingPerspective-based Reading

Roles• to be inspected • to conduct an inspection

Inspection

Process

Reading Techniques

• Organizer• Moderator• Inspector• Author• (Recorder)• (Presenter)

Products

Planing OverviewDetection Collection Correction

Follow-up





Slide 8

8 Summary8 Summary

1 Goals1 Goals





Benefits of Inspections: Early defect detection

Inspections help to detect more defects early in the development process Early Quality Improvement

15

30

15

25

30

15

15

25

45

10

20

30

5

10

10

0

10

20

30

40

50

60

70

80

90

Analysis Phase Design Phase Coding Module test System test System use

Defect detection according to phases without inspections

Code Defects

Design Defects

Requirements Defects

50

30

50

10

30

50

5

5

35

4

10

10

15

5

0

10

20

30

40

50

60

70

80

90

Analysis Phase Design Phase Coding Module test System test System use

Defect detection according to phases with inspections

Code Defects

Design Defects

Requirements Defects





Slide 9

8 Summary8 Summary

1 Goals1 Goals





Benefits of Inspections: Productivity and Cost

Inspections improve productivity by finding defects when they are cheaper to detect and correct Inspections increase productivity and lower cost

0

5

10

15

20

25

30

35

Analysis Phase Design Phase Coding Module Testing System Testing System Use

No

rmalized

Co

st

for

Defe

ct

Dete

cti

on

an

d R

em

oval

Requirements

Design

Code





Slide 10

8 Summary8 Summary

1 Goals1 Goals





Benefits of Inspections: Useful spin-off for participants

• Learning by experience

• Participants learn principles and and patterns of defect discovery.

they avoid these defects in their own work.

• Participants learn about good development practices.

they use them in their own work.

Inspections help integrate defect prevention processes into development processes!





Slide 11

8 Summary8 Summary

1 Goals1 Goals





Other Benefits of Inspections

In the long run...

• inspections convince participants to develop more understandable products

Products are easier to maintain.

• defect data allow

the improvement of the software development process in a project (root cause analysis).

the development of description, prediction, and evaluation models.





Slide 12

8 Summary8 Summary

1 Goals1 Goals





Importance of Inspection Evaluation

• Remove the NAH (not applicable here) symptom - prove that inspections are useful in a given organization

• Demonstrate the value of performing inspections in a given context

• Detect the weaknesses in the inspection approach and improve it (e.g. by introducing reading techniques)

• Help tailor inspections according to company and project specifics





Slide 13

8 Summary8 Summary

1 Goals1 Goals





Three Approaches for Inspection Evaluation

1. Comparison of a particular inspection to a baseline.Example: Statistical Process Control (SPC)

2. Comparison of different inspection types in a (quasi-) controlled experiment.Example: Comparing the effectiveness of checklist and Perspective-based reading.

3. Comparison of inspections to other activities that help detect defects, such as testingExample: Comparing the effectiveness of inspection and functional testing.

All approaches require measurement!





Slide 14

8 Summary8 Summary

1 Goals1 Goals





MeasurementWhat needs to be measured?• Defects

– Number detected by each inspector andin the meeting

– Classification of each defect (ODC)– Effectiveness estimates– ...

• Number and Effort of Participants– Number of inspectors– Effort for defect detection, collection

and correction per participant – ...

• Duration– Number of days between inspection activities– ...

• Characteristics of the inspected software product– Size (e.g., pages)– Complexity– ...





Slide 15

8 Summary8 Summary

1 Goals1 Goals





MeasurementHow to perform the evaluation?

1. Approach: Use control limits to compare a particular inspection with the ones performed previously.

Requirement: Process stability and baseline (~20 data points)





Slide 16

8 Summary8 Summary

1 Goals1 Goals





Measurement

How to perform the evaluation?

2. Approach: Compare different inspection types by performing a (quasi-)experimental comparison of inspection

– Effectiveness - % of defects detected– Efficiency - cost effectiveness - effort to

detect and to correct a defect– interval (duration)

Requirement: • Hypotheses regarding influential factors (e.g.

reading technique)

• A controllable context (e.g., a training situation)

• Total number of defects needs to be known

Expert judgement (subjective)

Capture-Recapture models (objective)

(Quasi-)experimental study (seeding)





Slide 17

8 Summary8 Summary

1 Goals1 Goals





MeasurementHow to perform the evaluation?

3. Approach: Perform a comparison between inspection and testing – Effectiveness - % of defects detected– Efficiency - cost effectiveness - effort to

detect and/or to correct a defect

Requirement: Inspection data, testing data, and evaluation model

test. costs - inspection coststesting costs + test. costs

Efficiency = 500 - 50500 + 500

== 0.45

Inspection

Testing

50

500 1000

Efficiency equals the proportion of total defect costs saved by implementing software inspections.





Slide 18

8 Summary8 Summary

1 Goals1 Goals





• to be inspected• to conduct an

inspectionRoles Inspection

Process

Reading Techniques

•Organizer•Moderator•Inspector•Author•Recorder•Presenter

Products

Planing OverviewDetection Collection Correction

Follow-up

“Inspection” - An Overview





Slide 19

8 Summary8 Summary

1 Goals1 Goals





Defect Detection

Question: How does one detect defects?

Answer: 1 By reading the document.

2 By understanding what the document describes.

3 By checking the required quality properties.

ProblemInspectors often do not know how to read the document!





Slide 20

8 Summary8 Summary

1 Goals1 Goals





Reading Documents

Reason • Developers learn how to write requirements, design, code, etc. But, they do not learn how to read them.

• Little technical support for the reading process is currently available.

Solution • Provide well-defined and tailored reading techniques.

Benefit • Increase the cost-effectiveness of inspections

• Provide models for writing documents of higher quality

• Reduce human influence on inspection results (i.e., ensure a more engineering approach in inspections)

ProblemInspectors often do not know how to read the document!





Slide 21

8 Summary8 Summary

1 Goals1 Goals





Definition and Types of Reading Techniques

What is it? A reading technique is a series of steps taken to examine a document. This allows a person to detect defects in a software product throughout the defect detection phase of an inspection.

What reading techniques for defect detection are available?

The following reading techniques for defect detection are available:• Ad-hoc• Checklists• Reading by Stepwise Abstraction• Defect-based Reading• Perspective-based Reading





Slide 22

8 Summary8 Summary

1 Goals1 Goals





Comparison of Reading Techniques

Technique Document Systematic? Focused? Controlledimprovement?

Tailorable? Trainingrequired?

Ad hoc Any No No No No NoCheck list Any Partially No Partially Yes PartiallyRSA Code Yes Yes No Partially YesDBR Requirem

entsYes Yes Yes Yes Yes

PBR Any Yes Yes Yes Yes Yes





Slide 23

8 Summary8 Summary

1 Goals1 Goals





PBR Concepts

• What is it? – Perspective-Based Reading (PBR) is a set of reading

techniques focusing on points of view.

– The idea is that a document has high quality if any potential stakeholder does not find a defect in it. Thus, a document should be read from various point of views (perspectives).

• Motivation:– make sure important perspectives are identified– make each inspector responsible for particular perspectives– allow inspectors to build up expertise in different aspects of

the document

• History: – Piloted at NASA/Goddard Space Flight Center

– Ongoing studies with students/professionals through universities: UMCP, UMBC, UNorway, ULund, Uni-Kl, …

– Adapted for industrial use/training by IESE





Slide 24

8 Summary8 Summary

1 Goals1 Goals





Perspectives - Example

• An example of perspectives: Each user of a Requirements document makes his/her demands upon it:

– a designer, who uses it to produce system design models;– a tester, who produces a test plan to ensure that the

system meets the requirements;– a user, who has to make sure the analysis documents

adequately capture the functionality needed in the final system;

Requirements Design Code Test Use

RequirementsDocument





Slide 25

8 Summary8 Summary

1 Goals1 Goals





Team inspection

• PBR assigns different perspectives to different inspectors, i.e., each inspector of an inspection team reads a document from a particular perspective.

• Benefits:

– An inspector focuses on specific defects (i.e., increase the defect detection probability of some defects) and does not have to check each and every detail.

– inspection team aims for increased defect coverage

Ad hoc coverage PBR coverage





Slide 26

8 Summary8 Summary

1 Goals1 Goals





Operational Scenarios

• PBR belongs to the scenario-based reading techniques.

• The ideas behind PBR are embodied within an operational scenario for each perspective.

– a procedural description of activities and questions that guides inspectors through the inspected documents

– represented as a set of steps to be followed

• Interpretation of “defect”

– “defect”: any quality property that is not fulfilled

– avoid focusing on correctness as the one and only quality property





Slide 27

8 Summary8 Summary

1 Goals1 Goals





Tailoring

• Summary so far: PBR is built on a number of concepts (perspectives, team inspection coverage, operational scenarios)

• Each of these concepts can be altered to conform to a particular organization:

– able to add new perspectives of interest to the organization

– able to change the mix of perspectives if I expect a predominance of defects of a certain type,

add more inspectors using relevant perspectives

– able to change the level of detail incorporated into the scenario according to the experience level of inspectors.

– PBR is tailorable to various documents (requirements, code) Also applicable in an object-oriented context!





Slide 28

8 Summary8 Summary

1 Goals1 Goals





Object-Oriented Inspections

Object-Oriented Development projects (particularly in the analysis and design phases) raise some additional issues regarding the inspected documents:

– Textual documents versus graphical ones (UML diagrams).– Information is distributed across various diagram types,

e.g., use-cases, class diagrams, state diagrams, activity diagrams.

– Diagrams may be created by case tools.– Large-scale software development projects result in a large

number of diagrams. They cannot be checked in one particular inspection.

How to determine the unit of analysis for inspectors?

– What is the “right” chunk of information for an inspection?

– What is the “right” set of documents/diagrams that describes the chunk of information?

How to provide reading support for inspectors?





Slide 29

8 Summary8 Summary

1 Goals1 Goals





• Organize an inspection around a logical entity (e.g., system, subsystem, component, class, operation) . Provide inspectors with all relevant documents/diagrams related to the logical entity.

The chunk of information is “intellectually manageable” and conceptually whole

Provides a solution for splitting large documents that cannot be reviewed in one inspection

• Use PBR to provide reading support. PBR can be tailored to support this organizational strategy.

Possible Solutions





Slide 30

8 Summary8 Summary

1 Goals1 Goals





Logical Entity

Requirements -Level

System1:1

Class DiagramOperationSchemata

StatechartDiagram

Class A

OO-Development OO-Development

n:1

StatechartDiagram

OperationSchemata

Class B

n:m

Code - Level

Function A

Function B

1:1

1:1

““Traditional” DevelopmentTraditional” Development





Slide 31

8 Summary8 Summary

1 Goals1 Goals





Reading Support

Class DiagramOperationSchemata

StatechartDiagram

Class A

• Unit of Analysis: Documentation of a class

• Goal for each participant:

– Analyze the documents provided for a class

– for the purpose of evaluation

– with respect to defect detection

– from the point of view of tester (or designer, …)

– in the context of a PBR inspection





Slide 32

8 Summary8 Summary

1 Goals1 Goals





Reading Support

Class diagram State Diagram Operation SchemataUser X XTester X XDesigner X X

Where to look?

• The tester needs to find out if the class does what it is supposed to do.

– “What it is supposed to do” is defined by the information contained in the state diagram and operation schemata

• Testing process:

– Derive test cases to mentally feed the class to be implemented with data

– determine the observed reaction

– observed reaction different from expected => defect!

How to check: Approach for a Tester





Slide 33

8 Summary8 Summary

1 Goals1 Goals





Reading Support

How to check? - Scenario (incomplete)Assume you are inspecting a class from the perspective of a tester. The main goal of a tester is to ensure the testability of the class. High quality thus corresponds to full testability. You will need to analyse test cases for the class, so if they are not available arrange for them to be created. A test case consists of a set of input values plus a set of output values and/or state changes expected for each combination of values. Follow the instructions below and answer the questions carefully.Locate the state diagram, the operation schema and the pseudo code description of the class under inspection. In the operation schema, identify the parameters which are preceded by the "supplied" keyword. Identify the equivalence classes for these parameters, and also the attributes named in the "reads" clause and document them. Using these equivalence classes, identify the minimal set of test cases needed to fully exercise the functional interface of the operation for the class. In the pseudocode description of the operation identify every conditional branch or loop which represents an execution branch. Identify an additional set of test cases which ensure that each state of the state diagram is covered. Mentally execute your test cases by feeding the state diagram and the operation schema with your test cases.While following the instructions answer the following questions• Do the branches in the pseudocode description match the condition outcomes in the operation schema?• Are all possible sets of input values properly addressed by the operation schema and the pseudocode description?





Slide 34

8 Summary8 Summary

1 Goals1 Goals





Empirical Results

Goal: Investigate the defect detection effectiveness of PBR and checklists when applied to UML-Design Documents.

Subjects: 18 Professionals with various degrees of programming experience (at least 1 year) in the context of an object- oriented development course.

Materials: - Two different design documents (Fusion-Process) - 3 Perspectives (Designer, Tester, Implementor) - 1 Checklist

Initial TeamResults:

Experimental Comparison

M in-M ax

25% -75%

M edian value

Differenc e in E ffec t ivenes s

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

P B R Chec k lis t





Slide 35

8 Summary8 Summary

1 Goals1 Goals





• Inspection is one of the few methods with empirically proven benefits.

• There are three approaches for evaluating inspections (statistical process control, (quasi-)experimentation with different inspection types, comparison of inspections and testing).

• Tailoring inspections to a particular environment involves the adaptation of the:– process– products– roles – reading techniques (often neglected)

• Perspective-based reading ensure a more systematic scrutiny by assigning specific viewpoints to inspectors and by providing procedural guidelines.

• PBR allows the additional challenges of object-oriented inspections (unit problem) to be tackled.

Summary





Slide 36

8 Summary8 Summary

1 Goals1 Goals





Bibliography

•Oliver Laitenberger, Colin Atkinson, Generalizing Perspective-based Inspection to handle Object-Oriented Development Artifacts, in Proceedings of the International Conference of Software Engineering, 1999

•Adam A. Porter, Lawrence G. Votta, and Victor R. Basili, Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, IEEE Transactions on Software Engineering, vol. 21, pp. 563-575, June 1995.

•Victor .R. Basili, Scott Green, Oliver Laitenberger, Fillipo Lanubile, Forrest Shull, Sivert Sorumgard, and Marvin V. Zelkowitz, The Empirical Investigation of Perspective-Based Reading, Journal of Empirical Software Engineering, vol. 2, no. 1, pp. 133-164, 1996.

•Victor R. Basili, Evolving and Packaging Reading Technologies, Journal of Systems and Software, vol. 38, July 1997.•Victor R. Basili, Forrest Shull, Fillipo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, 1999 (to appear).

•Carolyn B. Seaman and Victor R. Basili, Communication and Organization: An Empirical Study of Discussion in Inspection Meetings, IEEE Transactions on Software Engineering, vol. 24, pp. 559--572, June 1998.

•Forrest Shull, Developing techniques for using software documents: A series of empirical studies, PhD thesis, University of Maryland, 1998.

•Oliver Laitenberger and Jean-Marc DeBaud, An Encompassing Life-Cycle Centric Survey of Software Inspection, Journal of Systems and Software, 2000.

•Oliver Laitenberger, Khaled El Emam, Thomas Harbich, An Internally Replicated Quasi-Experimental Comparison of Checklist and Perspective-based Reading of Code Documents, IEEE Transactions on Software Engineering, 2000.

•Lionel Briand, Khaled El-Emam, Thomass Fussbroich, and Oliver Laitenberger, Using Simulation to Build Inspection Efficiency Benchmarks for Development Projects, in Proceedings of the Twentieth International Conference on Software Engineering, pp. 340-349, IEEE Computer Society Press, 1998.

•Lionel C. Briand, Khaled El Emam, Bernd Freimut, Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE Transactions on Software Engineering, 2000.

•Khaled El Emam, Oliver Laitenberger, Thomas Harbich, The Application of Subjective Estimates of Effectiveness to Controlling Software Inspections, Journal of Systems and Software, 2000.

A bibliography of more inspection-related literature can be found at: http://www.iese.fhg.de/ISE/Inspbib/inspection.html