copyright (c) 2003 howard e. dow1 results from inspecting test automation scripts howie dow...
TRANSCRIPT
![Page 1: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/1.jpg)
Copyright (c) 2003 Howard E. Dow
1
Results from Inspecting Results from Inspecting Test Automation ScriptsTest Automation Scripts
Howie Dow
![Page 2: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/2.jpg)
Copyright (c) 2003 Howard E. Dow
2
Agenda
Audience - Project - Terminology
What the team did
Results and data
Benefits, pitfalls and how to
Conclusions
![Page 3: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/3.jpg)
Copyright (c) 2003 Howard E. Dow
3
Car Cah
Parked Pahked
Rs ????Idea
Idear
![Page 4: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/4.jpg)
Copyright (c) 2003 Howard E. Dow
4
Target Audience
EngineersProject leaders
IdeallyTrained on inspectionsSome inspection experience
![Page 5: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/5.jpg)
Copyright (c) 2003 Howard E. Dow
5
Project
Call center system
Functional tests
GoalsDefect free test scripts
On time
Increase skill level
![Page 6: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/6.jpg)
Copyright (c) 2003 Howard E. Dow
6
Skill levelsSkill Level (Red=Available, Green=Needed)
0
0.5
1
1.5
2
2.5
3
3.5
1 2 3 4
Engineer Number
Ski
ll L
eve
l (1
=L
ow
, 2=
Me
diu
m,
3=H
igh
)
![Page 7: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/7.jpg)
Copyright (c) 2003 Howard E. Dow
7
Terminology
Inspection = Software Inspection = Formal Inspection
= “Fagan Style” Inspection
Reviews
Walk-Throughs
Any technique
![Page 8: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/8.jpg)
Copyright (c) 2003 Howard E. Dow
8
Terminology
Inspection characteristicsGoals
Rules, Roles, Responsibilities
Process
![Page 9: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/9.jpg)
Copyright (c) 2003 Howard E. Dow
9
Goals of Inspections
Find defects
Collect data
Archive data
Use data
Learn
![Page 10: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/10.jpg)
Copyright (c) 2003 Howard E. Dow
10
Etiquette, Customs and Rules
Focus on the product
Be prepared
Open mind
Language
![Page 11: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/11.jpg)
Copyright (c) 2003 Howard E. Dow
11
Roles and Responsibilities
Moderator Recorder
Reader Author
Management
Inspector
![Page 12: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/12.jpg)
Copyright (c) 2003 Howard E. Dow
12
Planning
ProductOverview
Preparation
Inspection
Rework
Data analysisand storage
Re-inspect(optional)
Inspection Process
![Page 13: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/13.jpg)
Copyright (c) 2003 Howard E. Dow
13
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
![Page 14: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/14.jpg)
Copyright (c) 2003 Howard E. Dow
14
Moderator Recorder
Reader Author Inspector
Manager
Roles and Responsibilities
![Page 15: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/15.jpg)
Copyright (c) 2003 Howard E. Dow
15
ProductOverview
Planning
Preparation
Inspection
Rework
Data analysisand storage
Re-inspect(optional)
Allowedsuggestions
Time not collected
Inspection Process Used
![Page 16: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/16.jpg)
Copyright (c) 2003 Howard E. Dow
16
Summary of key differences
Omit product overview
Involved management
Suggesting fixes allowed
Strong bias to re-examine after rework by another team member
Omission - collecting preparation time
![Page 17: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/17.jpg)
Copyright (c) 2003 Howard E. Dow
17
Ready to go
17 Scripts
3-6 inspectors
![Page 18: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/18.jpg)
Copyright (c) 2003 Howard E. Dow
18
Data CollectedScript identifier
Author
Date completed
Date inspected
Size
Defect: number, location, description
Inspection time
Inspectors
![Page 19: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/19.jpg)
Copyright (c) 2003 Howard E. Dow
19
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
![Page 20: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/20.jpg)
Copyright (c) 2003 Howard E. Dow
20
Results
No operational defects found in inspected test scripts
On time delivery
Increased skill level Defects
![Page 21: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/21.jpg)
Copyright (c) 2003 Howard E. Dow
21
Summary Data - 1
TotalsScripts inspected: 17
LOC inspected: 12,949
Defects found: 544
Time spent: 24.2 hours 75 hours
![Page 22: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/22.jpg)
Copyright (c) 2003 Howard E. Dow
22
Summary Data - 2
Number of Scripts Written
0
2
4
6
8
A B C
Engineer
![Page 23: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/23.jpg)
Copyright (c) 2003 Howard E. Dow
23
Summary Data - 3
Calculated Defect density: 42 Defects/KLOC
Inspection rate: 535 LOC/hour 173 LOC/hour
Defect find rate: 22.5 Defects/hour 7 Defects/hr
![Page 24: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/24.jpg)
Copyright (c) 2003 Howard E. Dow
24
Summary Data - 4
Severity type Quantity Percent of total
High 15 2.8%
Medium 209 38.4%
Low 320 58.8%
Total 544 100%
![Page 25: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/25.jpg)
Copyright (c) 2003 Howard E. Dow
25
Percent Severity
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%1 3 5 7 9 11
13
15
17
Inspection Date (earliest to most recent)
% Low PriorityDefects
% Medium PriorityDefects
% High PriorityDefects
![Page 26: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/26.jpg)
Copyright (c) 2003 Howard E. Dow
26
Inspectors per Inspection
0
1
2
3
4
5
6
7
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
# o
f In
sp
ec
tors
![Page 27: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/27.jpg)
Copyright (c) 2003 Howard E. Dow
27
Defects per KLOC
0.020.040.0
60.080.0100.0120.0
140.0160.0
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
De
fec
ts/K
LO
C
![Page 28: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/28.jpg)
Copyright (c) 2003 Howard E. Dow
28
Defects Found per Hour
0.0
5.0
10.0
15.0
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
![Page 29: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/29.jpg)
Copyright (c) 2003 Howard E. Dow
29
Defect Count vs. Size (LOC)
0
1020
3040
50
6070
8090
100
0 500 1000 1500 2000
Size (LOC)
De
fec
t C
ou
nt
![Page 30: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/30.jpg)
Copyright (c) 2003 Howard E. Dow
30
Size vs. Defects/KLOC
0
500
1000
1500
2000
2500
0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0
Defects/KLOC
Siz
e
![Page 31: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/31.jpg)
Copyright (c) 2003 Howard E. Dow
31
ObservationDefect Count vs. Size (LOC)
0
1020
3040
50
6070
8090
100
0 500 1000 1500 2000
Size (LOC)
De
fec
t C
ou
nt
![Page 32: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/32.jpg)
Copyright (c) 2003 Howard E. Dow
32
Observation
% Medium Severity Defects
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
![Page 33: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/33.jpg)
Copyright (c) 2003 Howard E. Dow
33
Observation
Size vs. Defects/KLOC
0
500
1000
1500
2000
2500
0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0
Defects/KLOC
Siz
e
???
![Page 34: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/34.jpg)
Copyright (c) 2003 Howard E. Dow
34
Size (LOC)
0
500
1000
1500
2000
2500
0 1 2 3 4 5 6 7 8
Size(LOC)Eng A
Size(LOC)Eng B
Size(LOC)Eng C
Observation
![Page 35: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/35.jpg)
Copyright (c) 2003 Howard E. Dow
35
Defects per KLOC
0.0
20.0
40.0
60.0
80.0
100.0
120.0
140.0
160.0
0 1 2 3 4 5 6 7 8
Defects/KLOCEng A
Defects/KLOCEngr B
Defects/KLOCEng C
Excellent!
Observation
![Page 36: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/36.jpg)
Copyright (c) 2003 Howard E. Dow
36
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
![Page 37: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/37.jpg)
Copyright (c) 2003 Howard E. Dow
37
Benefits
No operational defects reported
On time delivery
Value of process shown
Some increase in expertise
![Page 38: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/38.jpg)
Copyright (c) 2003 Howard E. Dow
38
Pitfalls
Must be
“Sloppy” data collection
Need for more training
![Page 39: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/39.jpg)
Copyright (c) 2003 Howard E. Dow
39
How to
Find a “critical” project
Suggest doing something different
Keep it simple - be flexible
Propose “Let’s put more eyes on the code.”
“Let’s inspect the code.”
![Page 40: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/40.jpg)
Copyright (c) 2003 Howard E. Dow
40
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
![Page 41: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/41.jpg)
Copyright (c) 2003 Howard E. Dow
41
Conclusions
Test scripts can be inspected
Share your resultsOther testers
Developers
Useful as a learning tool
Avoid unrealistic expectations
![Page 42: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/42.jpg)
Copyright (c) 2003 Howard E. Dow
42
ReferencesThe original article on software inspections. “Design and Code Inspections to Reduce Errors in Program Development”, Michael Fagan, IBM Systems Journal, Volume 15, No. 3, 1976. This may be available as reprint at: http://www.research.ibm.com/journal/si/382/fagan.pdf
Handbook of Walkthroughs, Inspections, and Technical Reviews, Daniel P. Freedman and Gerald M. Weinberg, 3rd Edition, Dorset House, 1990
Software Inspection, Tom Gilb and Dorothy Graham, Addison-Wesley, 1990
![Page 43: Copyright (c) 2003 Howard E. Dow1 Results from Inspecting Test Automation Scripts Howie Dow howie.dow@rcn.com hdow@alumni.carnegiemellon.edu](https://reader035.vdocument.in/reader035/viewer/2022062407/56649ca55503460f949672a7/html5/thumbnails/43.jpg)
Copyright (c) 2003 Howard E. Dow
43
Howie [email protected]
Thank you for your time.
Questions please