s oftware i nspection . q uality : testing, inspection, and analysis 1. testing is the most widely...
TRANSCRIPT
SOFTWARE INSPECTIONhttp://bzupages.com/
QUALITY: TESTING, INSPECTION, AND ANALYSIS
1. Testing is the most widely used approach to manage software quality
2. Testing and inspection typically account for more resource use than actual design and coding.
3. Testing and inspection cannot find all defects.4. Testing and inspection do not create quality.
Development practices create quality.5. Choices regarding testing and inspection are
influenced by quality metrics visible to management.
6. There are emerging assurance techniques that complement test and inspection
2
Author: M. Muzaffar Hameed Slides available on http://bzupages.com/
SOFTWARE INSPECTIONWhat are software inspections (reviews)? Meetings (real or virtual) during which designs and code are
reviewed bypeople other than the original developer.What are the benefits of inspections?New perspectiveFinding defects may be easier for people who haven't seen the
artifact before and don’t have preconceived ideas about its correctness.Knowledge sharingRegarding designs and specific software artifactsRegarding defect detection practices Find flaws earlyCan dramatically reduce cost of fixing themDuring detail design – even before code is writtenOr code that does not yet have a test harnessOr code in which testing has found flaws but root causes are not
understoodReduce rework and testing effortCan reduce overall development effort
3
Auth
or: M
. Muza
ffar H
am
eed S
lides a
vaila
ble
on
http
://bzu
pages.co
m/
INSPECTION VS TESTING
What attributes are well-handled by inspections but not testing?
“Fuzzy” non-functional properties Maintainability, evolvability, reusabilityOther properties tough to testScalability, efficiencySecurity, integrityRobustness, reliability, exception handlingRequirements, architecture, design documentsCannot “execute” these as a test
4
Auth
or: M
. Muza
ffar H
am
eed S
lides a
vaila
ble
on
http
://bzu
pages.co
m/
KINDS OF INSPECTION1. Inspections / Formal Technical ReviewsParticipation defined by policyDevelopersDesignated key individuals – peers, QA team, Review
Board, etc.Advance preparation by participantsTypically based on checklistsFormal meeting to discuss artifactLed by moderator, not authorDocumented process followedMay be virtual or conferencedFormal follow-up processWritten deliverable from reviewAppraise product
5
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
KINDS OF INSPECTION (CONT.)
2. WalkthroughsNo advance preparationAuthor leads discussion in meetingNo formal follow-upLow cost, valuable for education3. Other review approachesPass-around – preparation part of an inspectionPeer desk check – examination by a single
reviewer (like pair programming)Ad-hoc – informal feedback from a team
member
6
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
TRADEOFF AMONG INSPECTION TECHNIQUES
Formal technical reviews will find more bugsFord Motor: 50% more bugs with formal
processBut they also cost more
7
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
REVIEW ROLESWho are the stakeholders in inspection?1. ModeratorOrganizes reviewKeeps discussion on trackEnsures follow-up happensKey characteristicsGood facilitatorKnowledgeableImpartial and respectedCan hold participants accountable and correct
inappropriate BehaviorSeparate role from Recorder Who captures a log of the inspection process
8
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
REVEIW ROLES (READER)
2. Reader (different from author)Presents materialProvides points of comparison for author and other
team membersDifferences in interpretation provoke discussionReveals ambiguities
If author were to present, others might not mention that their interpretation is different
AlternativeGet comments section by sectionFaster, but does not capture differing perspectives as effectively 9
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
REVEIW ROLES (AUTHOR)
Describes rationale for workNot moderator or readerConflict between objectivity required of moderator/reader and advocacy for the author’s own workOthers raise issues more comfortablyNot recorderTemptation to not write down issues the author disagrees withSignificant benefits to attendingGain insight from others’ perspectivesCan answer questionsCan contribute to discussion based on knowledge of
artifactPotential downside: meeting may be confrontational
10
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
INSPECTION PROCESS
PlanningDetermine objectivesChoose moderatorIdentify inspectors
Good to involve people with connection to artifact
e.g. depends on, interfaces with.Schedule meeting(s)General guideline: 150-200 SLOC/hour, or 3-4 pages/hourPrepare and distribute inspection package
Deliverable, supporting docs, checklistsCross-reference specs, standards
11
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
INSPECTION PROCESS (CONT.)Overview meetingInformal meetingGoal: go over features, assumptions, background, contextOptional stageMay be able to use paper overview or shared context
PreparationInspectors examine deliverable
Defects: cause an error in the productNon-defects: improvements, clarification, style, questions
May want to list typos/spelling/format/style separately and not
discuss during the meetingConformance to standards & specificationOften use checklist
General guidelineprep time ~ meeting time
12
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
INSPECTION PROCESS (CONT.)The MeetingReader describes one segment at a time
Inspectors respond: defects, questions, suggestionsRecorder writes down each defect, suggestion, issueThis is the primary deliverableModerator
Avoid problem solving, inappropriate behavior, lack of participation At conclusion: prepares report with appraisal and data
Outcomes: Appraisal of productAccepted (minor changes, no follow up)Accepted conditionally (minor changes, verification)Reinspect following rework (major changes)Inspection not completed
Outcomes: Input on improving inspection process
Variant: reviewers make comments on electronic bulletin boardCost is lowerLose benefits of direct meeting (face to face, telephone)
Synergy - new bugs found (4%? 25%?)Learning by participantsCommunication about product
13
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
INSPECTION PROCESS (CONT.)Follow-up by authorAuthor addresses each item
Ensure understanding of issueIs it a defect or not? Is it a feature request or requirement change?Fixes defects and makes improvementsUncorrected/unverified defects go into defect tracking system
DeliverablesCorrected work productResponse to each issue and rationale for action
Moderator (or verifier) meets with authorCheck resolution of issuesExamine corrected deliverable
Author checks in code14
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
INSPECTION PROCESS (CONT.)AnalysisCausal analysis
Analyze root causes of defectsMake improvements to development and QA processes
Add issue to checklistChange testing approachDevelop or purchase new static analysis
Measuring effectivenessPercentage of bugs found during inspection
vs. found by other means or afterwards (test, customer)
Measuring efficiency“Defects per hour”Will decrease as your process improves
15
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
MEETINGS: REVIEW GUIDELINESBuild reviews into your scheduleOtherwise unexpected and viewed as intrusionRecognize that reviews can accelerate schedule by reducing other V&V
activitiesKeep review team smallGeneral guidelines: 3-7 participants3 is minimum for formal process to workBelow 3, too few perspectives besides authorAbove 7, work may be slowed by process, schedulingSmaller groups for code, larger groups for other documentsKnowledge is spread around more, more stakeholdersParticular for requirementsFind problems, but don't try to solve themTypically less expensive to address 1-on-1Guideline: halt solution discussion after 1-3 minutesLimit meetings to 2 hours maximumAttention span gets lost beyond thisRequire advance preparationProvides much of the value of a (formal) review 16
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
MEETINGS: CHECKLISTSBenefits Focus on likely sources of error Form quality standard that aids preparers Can bring up issues specific to a productShould be short About seven items
If more, group and do multiple passesFocus Priority issues Issues unlikely to be found other ways Historical problems Issues specific to the documentStart with checklist from well-known source Refine based on experience
17
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
PEOPLE: SOCIAL ASPECTS OF REVIEWSReviews are challenging Authors invest self-worth in product Encourages you to avoid letting others find errorsFor Authors Recognize value of feedback Place value in making code easy to understand Don’t take criticism of code personallyFor reviewers Don’t show off how much better/smarter you are Be sensitive to colleagues
Bad: "you didn't initialize this variable“Good: "I didn't see where this variable was initialized" 18
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/
WHAT TO INSPECT?Requirements, design documentsDifficult to validate in other waysMay have high associated risk
Especially important to get rightCheaper to fix earlier on in process
Many different perspectives are helpfulNeed involvement of multiple stakeholdersCritical or uncertain pieces of codeSecurity-critical codeSafety-critical codeStart inspections at the earliest stages of processCatch mistakes early, when easy to fixAllow rest of system to be built with knowledge gainedSample segments when there is a large body of workConsider what are good “coverage” criteria
19
Auth
or: M
. Muza
ffar H
am
eed S
lides
availa
ble
on h
ttp://b
zupages.co
m/