icpc12a.ppt

Professional statusvs. Expertise

Zephyrin Soh et al.

Problem andMotivations

Problem

Motivations

Related Work

Expertise Studies

UML Class DiagramComprehension

Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise

RQ3: Status vs. Expertise

RQ4: Question Precision

Conclusion andFuture Work

Conclusion

Threats to Validity andFuture Work

Professional Status and Expertise for UMLClass Diagram Comprehension:

An Empirical Study

Zephyrin Soh, Zohreh Sharafi, Bertrand Van den Plas,Gerardo Cepeda Porras, Yann-Gael Gueheneuc and

Giuliano Antoniol

Department of Computer and Software EngineeringEcole Polytechnique de Montreal, Quebec, Canada

June 13, 2012

Pattern Trace Identification, Detection, and Enhancement in JavaSOftware Cost-effective Change and Evolution Research Lab


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Outline

Problem and MotivationsProblemMotivations

Related WorkExpertise StudiesUML Class Diagram Comprehension

Empirical StudyStudy Design

ResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question Precision

Conclusion and Future WorkConclusionThreats to Validity and Future Work

2 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Problem and MotivationsProblem (1/1)

What is experience?

To manage subject/programmer experience:

I Years and education as main criteria [1]

I Authors sometime combine many criteria

[1] J. Feigenspan et al., Measuring Programming Experience, ICPC 2012, pp.73-82.3 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Problem and MotivationsMotivations (1/1)

MotivationsI Consider two following cases:

I A student who used UML for 4 years during her studyI A professional with 3 years of experience with UML

Who is the best at understanding of UML classdiagrams?

I Project managers when recruiting a new softwaredesigner by prioritized the important “factor”

I Future designers to know “where” to acquire thecompetitive skills by considering the important “factor”

4 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Related WorkExpertise Studies (1/3)

Previous work on expertise

I Novices spend less time than experts analysts [2]

I Graduate students are faster than junior ones andintermediate professionals [3]

I Experts are better for abstract questions and novices arebetter for concrete questions [4]

I Experts and novices have different program model fordocumentation task, no difference for reuse task [5]

[2] K. D. Schenk, N. P. Vitalari, and K. S. Davis, Differences between noviceand expert systems analysts: what do we know and what do we do?, Journalof Management Information System, vol. 15, pp. 9-50, June 1998

5 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion








[3] E. Arisholm and D. I. K. Sjøberg, Evaluating the effect of a delegatedversus centralized control style on the maintainability of object-orientedsoftware,IEEE Transactions on Software Engineering, vol. 30, no. 8, pp.521-534, aug. 20045 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion








[4] B. Adelson, When novices surpass experts: The difficulty of a task mayincrease with expertise,Journal of Experimental Psychology: Learning,Memory, and Cognition, vol. 10, pp. 483-495, Jul. 1984

5 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion








[5] J.-M. Burkhardt, F. Detienne, and S. Wiedenbeck, Object-orientedprogram comprehension: Effect of expertise, task and phase, EmpiricalSoftware Engineering, vol. 7, no. 2, pp. 115-156, 2002

5 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Comparison

To compare our work with previous work, we consider:

⇒ Object

⇒ Kind of task/question

⇒ Subject categorisation criterion

6 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Comparison

Ref.

[2] textual description requirementsanalysis

years of experiencerating scale ofsupervisors

[3] Java program change task students andprofessionals

[4] program +flowcharts

abstract +concrete question

undergraduatestudentsfellow teachers

[5] database program documentationand reuse

students and ex-perts (nominationby colleagues, ...)

Legend: Same to our work | Different to our work

[2] Schenk et al. (1998)[3] Arisholm and Sjøberg (2004)[4] Adelson (1984)[5] Burkhardt et al. (2002)

6 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Limitations

Previous work:I Did not precisely distinguish years of experience and

professionalism:I Inexperienced students as novices [5]I Senior professionals with less years of programming

experience than graduate students [3]

We distinguish the years of experience fromprofessionalism

I Studied the source code or textual descriptions ofrequirements

We use the UML class diagram

[5] Burkhardt et al. (2002)[3] Arisholm and Sjøberg (2004)7 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Related WorkUML Class Diagram Comprehension (1/2)

UML and eye-trackingI Stereotype, color, and layout facilitate class diagram

exploration and comprehension [6]

I Multi-cluster (by requirement) and three-cluster (bystereotype) layout positively affect the comprehension ofclass diagrams [7]

I Canonical representation of the Visitor pattern in classdiagram reduce the effort of maintenance task [8]

I The representations of design patterns affect theidentification of their participants and their roles [9]

[6] S. Yusuf, H. Kagdi, and J. I. Maletic, Assessing the comprehension of UMLdiagrams via eye tracking, ICPC’07[7] B. Sharif and J. I. Maletic, An empirical study on the comprehension ofstereotyped UML class diagram layouts, ICPC’09

8 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



UML and eye-trackingI Stereotype, color, and layout facilitate class diagram

exploration and comprehension [6]

I Multi-cluster (by requirement) and three-cluster (bystereotype) layout positively affect the comprehension ofclass diagrams [7]

I Canonical representation of the Visitor pattern in classdiagram reduce the effort of maintenance task [8]

I The representations of design patterns affect theidentification of their participants and their roles [9]

[8] S. Jeanmart, Y.-G. Gueheneuc, H. Sahraoui, and N. Habra, Impact of thevisitor pattern on program comprehension and maintenance, ESEM’09, Oct2009, pp. 69-78[9] G. Cepeda Porras and Y.-G. Gueheneuc, An empirical study on theefficiency of different design pattern representations in UML class diagrams,Empirical Software Engineering, vol. 15, no. 5, pp. 493-522, 20108 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Subjects’ categories

Previous work used subject’s proficiency as categorisationcriterion:

I Subjects’ performance in task realization

I Subjects’ grade in the course they were enrolled

MotivationsI No previous work that uses the maintenance task on

UML class diagrams and eye-tracking system to studyseparately the professional status and the expertise

I Combine expertise studies and UML eye-tracking studies

9 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Empirical StudyStudy Design (1/8)

Research QuestionsI RQ1: What is the relation between a designer’s

professional status and her class diagramcomprehension?

I RQ2: What is the relation between a designer’sexpertise and her class diagram comprehension?

I RQ3: What is the most important factor betweenexpertise and professional status?

I RQ4: What is the effect of the question precision onthe comprehension of a UML class diagram?

10 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Objects and Tasks

= ArgoUML, JUnit, and QuickUML

Number ofclasses/Interfaces

Averagenumber ofattributes perClass/Interface

Averagenumber ofmethods perClass/Interface

ArgoUML 10 0.4 8.6JUnit 14 0.57 6.14

QuickUML 16 1.75 3.87

= : one maintenance task per object

11 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Independent variables

= Professional status + Expertise

I Professional status= practitioners (9)(in industry)

= students (12)

I Expertise: We used the number of years of experienceto categorise experts and novices.

I Pair-wise Wilcoxon comparison (+ Bonferronicorrection)

I Categorization with the highlest Cliff’s δ value

= experts (12): {3, 4, 5} years of experience

= novices (9): {1, 2} years of experience

12 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Mitigating variable

Question precision: The level of details in the formulationof the question:

I Precise: state the kind of operation to perform(add/remove) and the kind of target element(class/method/attribute)

I Not precise: no operation or target element

13 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Dependent variables

I Accuracy, Time spent

I Search effort = convex hull & spatial density [10]

I Overall effort = AFD [9] and NRRF [8]

I Question comprehension effort = NDQA and NFQA

AFD: Average Fixation DurationNRRF: Normalized Rate of Relevant FixationsNDQA: Normalized Duration in Question AreaNFQA: Normalized Fixations in Question Area

[8] Jeanmart et al. (2009)[9] Cepeda Porras and Gueheneuc (2010)[10] J. H. Goldberg and X. P. Kotval, Computer interface evaluation using eyemovements: methods and constructs, Journal of Industrial Ergonomics, vol.24, no. 6, pp. 631-645, 199914 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



15 / 24

Convex hull areaI Smaller convex set of fixations

containing all subject’s fixations

I Smaller convex hull ⇒ close fixations⇒ less search effort


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



Spatial density

I Number of visited cells / total number of cells

I less visits ⇒ less search effort

I In TAUPE [11], cell’s size = 64x64px

[11] B. D. Smet, L. Lempereur, Z. Sharafi, Y.-G. Gueheneuc, G. Antoniol, andN. Habra, Taupe: Visualising and analysing eye-tracking data, Science ofComputer Programming, 201116 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



17 / 24

Overall effort: Fixations’ duration and relevance


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



17 / 24

Question Comprehension Effort: Fixations’ count and duration


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


ResultsRQ1: Status (1/1)

●

● ●●

● ●●

●

●

100 300 500 700

020

4060

8010

0

(a) Accuracy − Time tradeoff for PractitionersTime spent (s)

Acc

urac

y (%

)

● ●

●

●

●

●

●

●

●●

●●

150 250 350

020

4060

8010

0

(b) Accuracy − Time tradeoff for StudentsTime spent (s)

Acc

urac

y (%

)

Practitioners are more accurate than students

Students spent around 35% less time thanpractitioners

I No significant difference for other dependent variablesI Students could be more accurate if spending more time

18 / 24

What is the relation between adesigner’s professional status and herclass diagram comprehension?


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●

●

●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

● ●

●

●

●

●

●

●

●●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)

Practitioners are more accurate than studentsStudents spent around 35% less time than

practitioners

I No significant difference for other dependent variablesI Students could be more accurate if spending more time

18 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●

●

●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

● ●

●

●

●

●

●

●

●●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)


practitionersI No significant difference for other dependent variables

I Students could be more accurate if spending more time

18 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●

●

●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

● ●

●

●

●

●

●

●

●●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)


practitionersI No significant difference for other dependent variablesI Students could be more accurate if spending more time

18 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


ResultsRQ2: Expertise (1/1)

●

● ●●

● ●●●● ●● ●

100 300 500 700

020

4060

8010

0

(a) Accuracy − Time tradeoff for ExpertsTime spent (s)

Acc

urac

y (%

)

●

● ●

●

● ●

●

●●

150 250 350

020

4060

8010

0

(a) Accuracy − Time tradeoff for NovicesTime spent (s)

Acc

urac

y (%

)

Experts are more accurate than novices

Novices spent around 33% less time than expertsExperts have a more efficient ability to search

relevant elements than novicesI No significant difference for other dependent variablesI Novices could be more accurate if spending more time

19 / 24

What is the relation between adesigner’s expertise and her classdiagram comprehension?


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●●● ●● ●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

●

● ●

●

● ●

●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)

Experts are more accurate than novicesNovices spent around 33% less time than experts

Experts have a more efficient ability to searchrelevant elements than novices

I No significant difference for other dependent variablesI Novices could be more accurate if spending more time

19 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●●● ●● ●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

●

● ●

●

● ●

●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)

Experts are more accurate than novicesNovices spent around 33% less time than expertsExperts have a more efficient ability to search

relevant elements than novices

I No significant difference for other dependent variablesI Novices could be more accurate if spending more time

19 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●●● ●● ●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

●

● ●

●

● ●

●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)


relevant elements than novicesI No significant difference for other dependent variables

I Novices could be more accurate if spending more time

19 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion



●

● ●●

● ●●●● ●● ●

100 300 500 700

020

4060

8010

0


Acc

urac

y (%

)

●

● ●

●

● ●

●

●●

150 250 350

020

4060

8010

0


Acc

urac

y (%

)


relevant elements than novicesI No significant difference for other dependent variablesI Novices could be more accurate if spending more time19 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


ResultsRQ3: Status vs. Expertise (1/1)

Experts are more accurate than practitioners

Experts spent around 7% less time than practitionersI When considering expert subjects

I Experienced students are more accurate thanexperienced practitioners

I Experienced students spent around 37% less time thanexperienced practitioners

I The effects of expertise on accuracy and time dependon the status

20 / 24

What is the most important factorbetween expertise and professionalstatus?


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion




Experts spent around 7% less time than practitioners

I When considering expert subjects




20 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion




Experts spent around 7% less time than practitionersI When considering expert subjects




20 / 24



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


ResultsRQ4: Question Precision (1/1)

Question Precision

What is the effect of the question precision on thecomprehension of a UML class diagram?

I The accuracy of students benefits from precise questiondescription

I The accuracy of novices benefits from precise questiondescription

21 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Conclusion and Future WorkConclusion (1/1)

22 / 24

Status

Expertise

Experts vs. Practitioners


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Conclusion and Future WorkConclusion (1/1)

22 / 24

Status Expertise



Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


Conclusion and Future WorkThreats to Validity and Future Work (1/1)

Threats to Validity and Future Work

I Construct validity: We did not use all combination oftreatments for each system

I Conclusion validity: Practitioners from the samecompany + difficulty to find inexperienced practitioners(only one)⇒ Practitioners from other company

I Internal validity: We did not limit the time (fatiguebiais)⇒ Limit the experiment time to investigate how muchtime affect the subject’s accuracy

I External validity: Only three systems and small rangeof years of experience⇒ Use other systems

23 / 24


Zephyrin Soh et al.


Problem

Motivations

Related Work

Expertise Studies


Empirical Study

Study Design

Results

RQ1: Status

RQ2: Expertise




Conclusion


24 / 24

Thanks for your attention!

Status Expertise


The accuracy of students and novicesbenefits from precise question descriptions

icpc12a.ppt

Technology