presentation shigeru sasao_pairedcomparison_2
TRANSCRIPT
PAIRED COMPARISON: A USER PERSPECTIVEShigeru Sasao
Master of Software Engineering
Carnegie Mellon University
AGENDA
Introduction to Paired Comparison Experiment comparing Ad-hoc, Planning
Poker, and Paired Comparison Observation of Usage in a Project Points of Improvement New Version Using Incomplete Cyclic Design Conclusion
2
INTRODUCTION TO PAIRED COMPARISON
We want to estimate the size of the following objects.
We know that size of C is 10 units big.
A B
CD 3
4
Estimate the relative sizes for all pairs: I think A is 3 times as large as D. I think A is half the size of B. I think B is 6 times as large as D....
A B
CD
5
A B C D
A 1 0.5 2 3
B 2 1 0.5 6
C 0.5 2 1 2
D 0.3 0.17 0.5 1
This is the “Judgment Matrix”. Principal diagonal is always 1. The other shaded regions are reciprocals of
the un-shaded region. Does not have to be perfectly consistent,
since it is an estimate.
I think A is 3 times
bigger than D…
From the Judgment Matrix, we can calculate the estimates of the object size:
1. Calculate the geometric mean of each row.
2. Obtain the “ratio scale”.
3. Calculate the estimated size.
6
7
For our example,1. Calculate the geometric mean of each row.
2. Obtain the “ratio scale”.• Sum the geometric means
1.32 + 1.57 + 1.19 + 0.41 = 4.49• Divide geometric mean of the row by the sum
of the geometric mean.
A 1.32B 1.57C 1.19D 0.41
A 0.29B 0.35C 0.26D 0.09
8
3. Calculate the estimated size.• We know from the beginning that C has size
of 10 units. • So, the reference value is C with ratio scale
entry of 0.26.
A 0.29/0.26 * 10 = 11.15 units
B 0.35/0.26 * 10 = 13.46 units
C 0.26/0.26 * 10 = 10.00 units
D 0.09/0.26 * 10 = 3.46 units
INTRODUCTION TO PAIRED COMPARISON
What did we do? We knew the size of 1 object (reference size). We compared pairs of relative sizes, and entered
it into a judgment matrix. Calculated the estimates for the objects.
The method reduces individual judgment errors by requiring multiple pair-wise comparisons of relative values.
9
EXPERIMENT COMPARING AD-HOC, PLANNING POKER, AND PAIRED COMPARISON
Comparison was made between ad-hoc methods, planning poker and paired comparison.
Study conducted among students from the Master of Software Engineering program at Carnegie Mellon.
Students have strong technical background with two to three years of industry experience.
10
EXPERIMENT SETUP
Students divided into three groups, with each group using either ad-hoc, planning poker, or paired comparison.
Estimations conducted in pairs. Five pairs per estimation technique. Students were asked to estimate the size
(Lines of Code) of different data structures (stack, queue, binary tree, etc).
Students were told that the size of “linked list (a)” was 40 LOC.
11
EXPERIMENT RESULT
12
EXPERIMENT RESULT
Planning poker and paired comparison show a vast improvement in precision over ad-hoc methods.
This result is consistent with previous reports by Miranda.
13
Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.
EXPERIMENT RESULT
Comparison of standard deviation between planning poker and paired comparison:
14
Data Structure Std. Dev.(Planning Poker)
Std. Dev. (Paired Comparison)
Stack 9.8 2.4Queue 11.4 4.9Binary Tree 25.5 13.3String Manipulation 31.6 8.6Linked List (b) 42.4 5.5Balanced Tree 56.0 21.4Hash Table 33.1 31.5
EXPERIMENT RESULT
Paired comparison produced more consistent estimations among estimators as compared to ad-hoc and planning poker.
Earlier studies by both Miranda and Shepperd show that paired comparison produces more reliable estimations than ad-hoc estimation methods.
15
OBSERVATION OF USAGE IN A PROJECT In the Master of Software Engineering
program, a team of four to five members work for an external customer.
16 months with a total resource of 4608 to 5760 person hours.
Observed the usage of paired comparison by a MSE team.
The project was to produce an integrated development environment to support global distributed teams.
Estimation of effort for architectural components prior to the detailed design/implementation stage. 16
OBSERVATION OF USAGE IN A PROJECT
Five team members and observers gathered in a conference for the estimation session.
A list of 12 components was prepared beforehand.
Pair-wise comparison between all 12 components, for a total of 66 comparisons.
For each comparison, team discussed size, complexity, number of state transitions to agree on a relative size.
Values entered into a spreadsheet tool.
17
OBSERVATION OF USAGE IN A PROJECT
Snapshot of the resulting judgment matrix:
Total of 66 comparisons in 54 minutes.
18
Reference Value
Artifact Name
Dat
a E
lem
, Mgr
Trac
eabi
lity
cont
rolle
r
Impo
rt U
I
Trac
. Vie
w C
ont
Dat
a U
I
Trac
eabi
lity
UI
Trac
eabi
lity
Ser
ver
AD
T D
ef
Tran
sf. C
hain
Impo
rt M
gr
Trac
eabi
lity
Con
f
RTC
Bri
dge
Ratio Scale
Estimated Value
Data Elem, Mgr 2.0 9.5 4.1 4.5 4.3 5.5 5.6 7.0 9.0 8.0 1.5 0.256 258.7Traceability controller 7.0 1.8 2.5 2.2 2.0 2.2 5.5 6.5 6.0 0.5 0.135 136.5
20 Import UI 0.5 0.2 0.2 0.4 0.4 0.7 0.8 0.9 0.1 0.020 20.0Trac. View Cont 2.0 1.8 1.5 1.6 2.2 3.5 3.0 0.3 0.076 76.7Data UI 0.8 0.5 0.6 0.8 2.8 2.4 0.4 0.050 50.8Traceability UI 0.8 0.7 2.5 3.8 3.5 0.4 0.067 67.4Traceability Server 1.1 2.0 3.5 3.2 0.4 0.067 67.9ADT Def 2.2 3.7 3.4 0.4 0.067 67.5Transf. Chain 2.5 2.8 0.3 0.038 38.4Import Mgr 1.6 0.1 0.021 21.0Traceability Conf 0.1 0.020 20.4RTC Bridge 0.182 183.7
POINTS OF IMPROVEMENT
1. Estimators became noticeably tired during the 54 minute estimation session.
2. As estimators became tired, they started extrapolating the new relative size values from old ones that have already been determined.
“Since A was 3 times the size of C and C was 2 times the size of D, A must be 6 times the size of D…”
19
NEW VERSION USING INCOMPLETE CYCLIC DESIGN To consider the estimators’ stamina, we need
to reduce the number of comparison. Use incomplete cyclic designs to reduce
comparisons.
Maintains balance and connectedness. 20
NEW VERSION USING INCOMPLETE CYCLIC DESIGN
Control the number of comparisons by the replication factor.
Example for a 10 component comparison:
21
Replication Factor Number of Comparisons10 458 406 304 202 10
NEW VERSION USINGINCOMPLETE CYCLIC DESIGN
Full comparison of 8 components (28 comparisons)
Replication factor of 4 (16 comparisons)
22
A B C D E F G HABCDEFGH
A B C D E F G HABCDEFGH
NEW VERSION USINGINCOMPLETE CYCLIC DESIGN
The reduction of comparison is made possible through: Incomplete cyclic design (ICD) Imputing missing values as the geometric mean
of comparisons made Studies have shown that very low replication
factors still produce reliable estimates, although precision may diminish as number of comparisons decrease.
23
NEW VERSION USING CYCLIC DESIGN
Developed a tool which uses the new version with cyclic design:
24
Replication factor
Randomized
comparison order
INTEGRATION WITH COCOMO
Use paired comparison to estimate LOC size as input into COCOMO.
25
CONCLUSION
Paired comparison produces more consistent estimations than ad-hoc methods and planning poker.
Can be used to directly estimate effort, or to estimate LOC size for input into parametric models such as COCOMO.
New version using incomplete cyclic designs reduce the number of comparison.
If you are interested in the tool, please contact me.
26
REFERENCES [Aguaron et al. 2003] Aguaron, Juan, Moreno-Jimenez, Jose Maria, The Geometric
Consistency Index: Approximated Thresholds, European Journal of Operational Research, 147 (2003), Pages 137-145.
[Crawford, Williams 1985] Crawford, Gordon, Williams, Cindy, The Analysis of Subjective Judgment Matrices, Rand Corporation, 1985.
[Miranda 2000] Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.
[Miranda 2001] Miranda, Eduardo, Improving Subjective Estimates Using Paired Comparison, IEEE Software, 2001.
[Miranda et al. 2009] Miranda, Eduardo, Bourque, Pierre, Abran, Alain, Sizing User Stories Using Paired Comparisons, Information and Software Technology Volume 51, Issue 9, September 2009, Pages 1327-1337, Butterworth-Heinemann, 2009.
[PlanningPoker] Cohn, Mike, Planning Poker, www.planningpoker.com, Mountain Goat Software.
[Shepperd et al. 2001] Shepperd, Martin, Cartwright, Michelle, Predicting with Sparse Data, IEEE Transactions on Software Engineering, VOL. 27, NO. 11, 2001.
[Spencer 1982] Spencer, Ian, Incomplete Experimental Designs for Multidimensional Scaling, Chapter 3. In R.G. Golledge and J.N. Rayner (Eds.), Proximity and Preference: Problems in the Multidimensional Analysis of Large Data Sets, University of Minnesota Press, 1982.
[Spencer 1983] Spencer, Ian, Monte Carlo Simulation Studies, Applied Psychological Measurement Vol. 7, No. 4, 1983.
27