8/12/2019 Lecture 16a Walkthroughs

1/59

Software Reviews,

Walkthroughs, Inspections, and

AuditsBetter to find an error twice than

never to find it at all.

[Freedman/Weinberg 90] Freedman,

D.P., G.M. Weinberg, "Handbook of

Update terminology

according

Give references

to ONeil

8/12/2019 Lecture 16a Walkthroughs

2/59

Why Do We Have Formal

Technical Reviews?

1. To err is human.2. Lots of errors escape the originator more

easily than anyone else.

3. Reviews are educational.

8/12/2019 Lecture 16a Walkthroughs

3/59

Purpose: Provide

visibility into state of project

opportunities for personnel (project and

non-project) to discuss topics related to

project

intermediate milestones and sense of

progress

assessment of technical adequacy of project

8/12/2019 Lecture 16a Walkthroughs

4/59

Use the power of a team to

Point out needed improvements of a

product.

Confirm the parts in which improvement is

not needed or desired.

Make technical work more uniform and

predictable in quality, which makes it moremanageable.

8/12/2019 Lecture 16a Walkthroughs

5/59

Technical review is different

from other reviews For example, budget and management

reviews

Technical review answers question: will this

product do the job its supposed to do

If answer is no, then

no schedule is on time, and

no cost is cheap enough.

8/12/2019 Lecture 16a Walkthroughs

6/59

When?

Single review that occurs on a particular date

Single review that occurs in response to a

particular condition Multiple reviews that occur periodically: e.g.,

monthly

Multiple reviews that occur in response to aparticular condition, e.g., code review forsubsystem

8/12/2019 Lecture 16a Walkthroughs

7/59

Benefits

10X reduction in errors in products

50-80% cost reduction

8/12/2019 Lecture 16a Walkthroughs

8/59

8/12/2019 Lecture 16a Walkthroughs

9/59

Why not just pass it around and

have reviewers sign off

Comments are too large Comments are unread

Reviewers dont have to take responsibility

for the content.

8/12/2019 Lecture 16a Walkthroughs

10/59

Requirements

Team participation

Documented procedures

Documented results

8/12/2019 Lecture 16a Walkthroughs

11/59

Types of Reviews

Management Review

Technical Review

Walkthrough

Inspection

Audit

8/12/2019 Lecture 16a Walkthroughs

12/59

8/12/2019 Lecture 16a Walkthroughs

13/59

8/12/2019 Lecture 16a Walkthroughs

14/59

Types of Reviews

Management Review

Technical Review

Focus on specification and design

Determine whether products meet specs

Walkthrough

Inspection

Audit

8/12/2019 Lecture 16a Walkthroughs

15/59

Scenario for a typical review

Leader: Welcome. We are here to review X for the

purpose of Y.

Each person is introduced. The recorder is introduced.

An agenda is presented and the review process

explained.

The list of material in the review packet is stated.

The review begins

8/12/2019 Lecture 16a Walkthroughs

16/59

Review

Agenda set by leader, agreed to by

committee

Review proceeds line-by-line.

Reviewers bring up issues as they encounter

them.

Issues are added to an Issues List.

Agenda changes as issues are encountered.

8/12/2019 Lecture 16a Walkthroughs

17/59

Round Robin Review

Force every member to participate.

Each person takes the lead for a section:

paragraph,

line of code,

function,

test case

Good way to get involvement from team.

8/12/2019 Lecture 16a Walkthroughs

18/59

Types of Reviews

Management Review

Technical Review

Walkthrough Used to find anomalies and evaluate conformance

Typically used for source code

Effective form of education

Inspection

Audit

8/12/2019 Lecture 16a Walkthroughs

19/59

Walkthroughs

Producerguides review

Step by step presentation of product:

Code, design, report, test cases Can cover lots of material

Can have more people attend, less prepared

More work for presenter

May be difficult to control interactions

Too many interruptions

Always hard to have a producer present

8/12/2019 Lecture 16a Walkthroughs

20/59

Walkthroughs

Participant should spend 1 hour preparing

for each hour in meeting

Should be scheduled to be brief

Should only review completed code or

document

Should require participants to sign report

8/12/2019 Lecture 16a Walkthroughs

21/59

Types of Reviews

Management Review

Technical Review

Walkthrough

Inspection

More formal type of walkthrough

Used to identify specific errors or problems

Audit

8/12/2019 Lecture 16a Walkthroughs

22/59

8/12/2019 Lecture 16a Walkthroughs

23/59

Types of Reviews

Management Review

Technical Review

Walkthrough

Inspection

Audit

Independent evaluation of process or product

Ensures that the process is being followed

8/12/2019 Lecture 16a Walkthroughs

24/59

Review Teams

Permanent or semi-permanent teams

IV&V or QA teams

Get to be very good at reviews

Need to have some way of reviewing the

reviewers.

8/12/2019 Lecture 16a Walkthroughs

25/59

The team Leader

Job: obtain a good review or report why it wasnot possible

Reflects the quality of the review process,not the quality of the product

Must be accurate

Recorder

Provide info for accurate report of review

Short, public notesCapture essence of each issue

Must ensure group has reached conclusion

Dont video tape

8/12/2019 Lecture 16a Walkthroughs

26/59

Tasks of leader

Monitor preparedness of team members

Set pace

Keep meeting on track Poll members to reach consensus

Ensure participation

Has right to terminate review if it is unproductiveDisagreements

Fatal flaws

8/12/2019 Lecture 16a Walkthroughs

27/59

Consensus

Decision of the group is equal to the most

severe opinion of the group members.

Be conservative.

Accept the doubts of the most doubting

member.

No I told you so, but I got outvoted.

8/12/2019 Lecture 16a Walkthroughs

28/59

Reviewers

Must be qualified to contribute

Must be unbiased

Dont invite management if it causes

conflict

The point is to review the project, not the

producers.

8/12/2019 Lecture 16a Walkthroughs

29/59

Rules for Reviewers

I. Be prepared

II. Evaluate product, not people.

III. Watch your language.

IV. One positive, one negative.

V. Raise issues, dont resolve them.

VI. Record all issues in public.

VII. Stick to technical issues.

8/12/2019 Lecture 16a Walkthroughs

30/59

Rules for Reviewers

If you find something, assume its a mistake.These are your peers, not your enemy. Remember

people are involved

Avoid why did you why didnt you say

instead I dont understand Why did you set the upper bound to 10 here?

I dont understand why the upper bound is 10 here.

seems trivial, but its not.

Do not get bogged down in style issues. Forexample, if efficiency isnt an issue, then dontmake it one.

If it makes things less maintainable, thats an issue.

If there are standards, then either stick to thestandard, or dispose of the standard.

8/12/2019 Lecture 16a Walkthroughs

31/59

8/12/2019 Lecture 16a Walkthroughs

32/59

Time

At most 2 hours at a time

Come back again if necessary

Depends on

Complexity

Size of project

Closeness to completion

8/12/2019 Lecture 16a Walkthroughs

33/59

Everyone must prepare!

Review packet

Everything relevant to making correct

judgment: code, specs, test data, results,standards, designs,

80% of failures stem from unprepared teams

8/12/2019 Lecture 16a Walkthroughs

34/59

Tactics for participation

Devils Advocate

One person makes strongest case against aproduct

Job is to find fault

Needs to be an actor Debugging

Producers leave known bugs in

Quality of review depends on how many found

Alarms

Time each persons contribution, cut off

Stand-up Reviews

Talk as long as you stand on one leg

8/12/2019 Lecture 16a Walkthroughs

35/59

Report

Goal: Does the product do the job its supposed todo?

Complete.

Correct.

Dependable as basis for other work

Measurable for purposes of tracking.

What was reviewed? Signatures of reviewers.

Leader and recorder.

8/12/2019 Lecture 16a Walkthroughs

36/59

Notes

Requires learning: to review and to write

Need to allocate time

Not a form of project management

But can provide information for tracking

Review early and often

But not too often

8/12/2019 Lecture 16a Walkthroughs

37/59

Need for Reviews

Reviews help organization deliver qualityproduct

Examine individual parts early in processIdentify problems spanning multiple items or

phases

Reviews help identify problems early

Reviews help build image of the productand the vision of the product in the minds ofthe team members

8/12/2019 Lecture 16a Walkthroughs

38/59

Cost of Software Reviews

Up-front cost for Training

Staff preparation

Review Conduct

Code review may be 8-15% of total cost of project

Offset by savings: reduced rework later in project(25-35% of total project cost saved) Raytheon: pre-1988: 43% of software cost in defect

correction 1994 (after software reviews instituted): 5% of project

cost in defect correction

8/12/2019 Lecture 16a Walkthroughs

39/59

Writing Issues

Groups of 3

Ill give you an issue from an issue list,

You tell me whats wrong with it.

Issues (and responses) fromHandbook of

Walkthroughs, Inspections, and Technical

Reviews, Daniel Freedman and Gerald

Weinberg, 3rdEd, Dorset House.

8/12/2019 Lecture 16a Walkthroughs

40/59

Writing Issues #1

Some of the explanations of user commands

were misinterpreted by members of the

review committee.

8/12/2019 Lecture 16a Walkthroughs

41/59

Comments on Issues #1

Not specific enough. Whichexplanations were

misinterpreted? Dont make the producers

guess, or they may change the wrong thing.

8/12/2019 Lecture 16a Walkthroughs

42/59

8/12/2019 Lecture 16a Walkthroughs

43/59

Comments on Issues #2

In writing, clarity isnt the most important

thing: its the onlything. If you dont want

to be not understood, dont never use nodouble negatives.

Another thingwhat is the issue anyway? Is

the maximum too low? Too high? Or is itthat some of the codes should have been

made not eligible? Or eligible? Be direct.

8/12/2019 Lecture 16a Walkthroughs

44/59

Writing Issues #3

The referenced table of constant values was

not part of the review packet.

8/12/2019 Lecture 16a Walkthroughs

45/59

Comments on Issues #3

Always give the most direct reference

available. There could be more than one

table, now or in the future. Why make theproducers search?

8/12/2019 Lecture 16a Walkthroughs

46/59

Writing Issues #4

The method used for maintaining the message

queue seems to solve a severe performanceproblem weve been having with the

production version of the RKY system.

8/12/2019 Lecture 16a Walkthroughs

47/59

Comments on Issues #4

This is an interesting observation, clearly and

directly stated. But what does it have to do

with the product under review? It might beworth millions, but it doesnt belong on the

Issues List for this product. It should go on

the Related issues list.

8/12/2019 Lecture 16a Walkthroughs

48/59

More issues #5The price/performance table should be sorted

using either Quicksort or Shell sort.

8/12/2019 Lecture 16a Walkthroughs

49/59

Comments #5

Raise issues, but avoid all temptation to giveadvice in the Issues List. They wont be

welcome if they come in that form, so ifyou really must give advice, find someunofficial vehicle. Take the producers tolunch, or out for a beer, before you share

your vast experience on matters of sorting.If your idea isnt worth the price of a beer,why not forget it?

8/12/2019 Lecture 16a Walkthroughs

50/59

More issues #6

The three diagrams drawn by Harold Mitter

are not in the standard format (DS-109)

required for such diagrams in our

installation.

8/12/2019 Lecture 16a Walkthroughs

51/59

Comments #6

This point is nicely specific, but why do we

have to mention poor Harold? We are

reviewing the product, not the people. Findanother way to identify the diagrams and

leave peoples names out of the Issues List.

8/12/2019 Lecture 16a Walkthroughs

52/59

More issues #7

The committee was unable to understand the

significance of paragraph 3.1.

8/12/2019 Lecture 16a Walkthroughs

53/59

Comments #7

Nothings wrong with this one. Its specific aboutwhich paragraph is under discussion, and it saysthe committee doesnt understand it. Who can

argue with that? Yet, surprisingly, people seem tobe afraid to express issues this wayWe dontunderstand Dont worry about being thoughtstupid.

If you dont understand it, its at least a potentialissue in documentation. And besides, lack ofunderstanding may mean there is somethingdreadfully wrong.

8/12/2019 Lecture 16a Walkthroughs

54/59

Yet more issues #8

The bubble sort used for sorting the table of

price/performance figures is a stupid

approach if the table should grow any

bigger than the present 20 entries.

8/12/2019 Lecture 16a Walkthroughs

55/59

Comments #8

The word stupid doesnt add anything at all and

might antagonize the producers. Take it out. Then

try to express factually and quantitatively why the

method is inappropriate for larger tables.

And if the approach isstupid, what of it? As Arthur

C. Clarke expressed it, It has yet to be proved

that intelligence has any survival value.

8/12/2019 Lecture 16a Walkthroughs

56/59

Yet more issues #9

If the BY NAME option is not used, the

structuring of the structure operands must

be equivalent to the structuring of the

structures in the arrays of structures.

8/12/2019 Lecture 16a Walkthroughs

57/59

Comments #9

This wasnt really taken from an Issues List,

but from an old PL/I manual. Still, Ive read

real issues that were almost as obscure. Idadvise you to follow the KISS axiom

(Keep It Simple, Stupid), except that I

would be contradicting my own advice fromthe previous question.

8/12/2019 Lecture 16a Walkthroughs

58/59

Yet more issues #10

Frieda Sonntag has no computer science

background, and her experience with PL/Iis nil. She should never have been

assigned the coding of this module.

8/12/2019 Lecture 16a Walkthroughs

59/59

Comments #10

Its non of the business of the review

committee who management has assigned

to particular jobs. The committees businessis to review the product and tell what state

its in. How it got to that state is another

issueand not for the Issues List. Who isresponsible is even less of an issue, and

raising it is sure to be ineffective.