grasp – overview of program comprehension studies - slide 1 auburn university computer science and...

GRASP – Overview of Program Comprehension Studies - Slide 1Auburn UniversityComputer Science and Software Engineering

The GRASP Research Project

An Overview of Program Comprehension Studies


Software Visualization

• Intuitively beneficial in comprehension tasks

• Empirical evidence is mixed

• Graphical representations of software are inherently useful, though particular representations may not be.


Control Structure Diagram (CSD)

• Visually depicts the control structure and module-level organization of source code

• A value-added feature of source code: Appears as a companion to the source code without disrupting its familiar appearance

Compact, intuitive, non-disruptive


CSD

[Adapted from Barnes, Programming in Ada 2nd Ed., Addison-Wesley, 1984]

task body TASK_NAME is begin loop for p in PRIORITY loop select accept REQUEST(p) (D : DATA) do ACTION (D); end; exit; else null; end select; end loop; end loop; end TASK_NAME;


CSD Æ¹¹¹¹¹¹¹¹çèétask body TASK_NAME isÊ¹Ë¹¹¹¹¹¹¹¹ §begin ¨¹¹®loop § 7¹¹±for p in PRIORITY loop § 5 7¹²´select¹¹Ã 5 5 6§ ¬¹¹¹¹¹¹¹¹¹ § 5 5 6¨êëìaccept REQUEST(p) (D : DATA) do § 5 5 6§ ªË¹¹¹¹¹¹¹¹ § 5 5 6§ ¨¹¹ ACTION (D);Â¹Ä 5 5 6§ ©end; § 5Â¹Ç 6¾¹¹ exit; § 5 5 ¶´else § 5 5 ¸¾¹¹ null; § 5 5 È end select; § 5 °end loop; § °end loop; ©end TASK_NAME;


CSD repeat := true; WHILE repeat LOOP text_io.get_line (item => response, last => last); -- Interpret user command IF (last = 1) THEN IF (response(1) = 'r') THEN text_io.new_line; flag := continue; repeat := false; exit Process_Loop; ELSIF (response(1) = 's') THEN -- Status check requested flag := status; repeat := false; ELSIF (response(1) = 'c') THEN -- Checkpoint interval change requested LOOP BEGIN


§ 5 6 § 6 ¨¹¹ repeat := true; § 5 6 § 6 ¨¹¹±WHILE repeat LOOP § 5 6 § 6 § 5 § 5 6 § 6 § 7¹¹ text_io.get_line (item => response, last => last); § 5 6 § 6 § 5 § 5 6 § 6 § 5-- Interpret user command § 5 6 § 6 § 7¹³ÍF (last = 1) THEN § 5 6 § 6 § 5 6¾¹³ÍF (response(1) = 'r') THEN § 5 6 § 6 § 5 6 6§ § 5 6 § 6 § 5 6 6¨¹¹ text_io.new_line; § 5 6 § 6 § 5 6 6¨¹¹ flag := continue; § 5 6 § 6 § 5 6 6¨¹¹ repeat := false; §Â¹Ç 6 § 6 § 5 6 6¾¹¹ exit Process_Loop; § 5 6 § 6 § 5 6 ³ÉLSIF (response(1) = 's') THEN § 5 6 § 6 § 5 6 6§ § 5 6 § 6 § 5 6 6§-- Status check requested § 5 6 § 6 § 5 6 6¨¹¹ flag := status; § 5 6 § 6 § 5 6 6¾¹¹ repeat := false; § 5 6 § 6 § 5 6 6 § 5 6 § 6 § 5 6 ³ÉLSIF (response(1) = 'c') THEN § 5 6 § 6 § 5 6 6§ § 5 6 § 6 § 5 6 6§-- Checkpoint interval change requested § 5 6 § 6 § 5 6 6¨¹¹®LOOP § 5 6 § 6 § 5 6 6§ 7¹¹´BEGIN

CSD


gtcand((rtoffinf[CANDSTART]+k-1),report,10,report,36); ivote = vote[rtoffinf[CANDSTART]+k-1]; if (ivote < 0L) ivote = 0L; cvicl(ivote,report,44,7); x430: cprint(flright); /* if (rtoffinf[DUALLOCATION] <= 0) continue; gtdcnd((rtoffinf[DUALSTART]+k-1),report,12); cprint(flright); */ /* loop back for next candidate */ } ; goto x455; /* no candidates */ x460: cprint(flright); smove(report,11,"*** Warning *** No candidates",0,29); cprint(flright);

CSD


§ 5 7¹¹ gtcand((rtoffinf[CANDSTART]+k-1),report,10,report,36); § 5 7¹¹ ivote = vote[rtoffinf[CANDSTART]+k-1]; § 5 7¹³´if (ivote < 0L) § 5 5 ¶¾¹¹ ivote = 0L; § 5 7¹¹ cvicl(ivote,report,44,7); § 5 5 § 5¹¹!x430: § 5 7¹¹ cprint(flright); § 5 5 /* if (rtoffinf[DUALLOCATION] <= 0) § 5 5 continue; § 5 5 gtdcnd((rtoffinf[DUALSTART]+k-1),report,12); § 5 5 cprint(flright); */ § 5 5 § 5 5 /* loop back for next candidate */ § 5 °} § 7¹¹ ;Â¹Ä 7¹¹ goto x455; § 5 § 5 /* no candidates */ §¹¹!x460: § 7¹¹ cprint(flright); § 7¹¹ smove(report,11,"*** Warning *** No candidates",0,29); § 7¹¹ cprint(flright);

CSD


Evaluation

• Subjective study: performance characteristics– Before GRASP was developed and distributed– CSD v. similar visualizations– rated according to 11 performance characteristics– significant preference for the CSD in 8 of 11

• Student preference surveys– After GRASP was in use in the AU CSSE curriculum

• Instrumentation– 3-4 years of AU utilization data

• Cumulative implication of these initial evaluations plus anecdotal evidence from industry provided motivation for continued development and study.


Empirical Research

• Two major research objectives– Evaluate and refine the CSD (and other visualizations such as

architecture diagrams as the research progresses).– Study how people use software visualizations in program

comprehension tasks.• Three major questions to address

– How do visualizations such as the CSD affect program comprehensibility?

– How do people use visualization techniques as tools in their work?

– How can the comprehensibility of a program be measured in an effective way?

• Funding provided by the National Science Foundation(EIA-9806777).


Controlled Experiments

CSD v. Plain Text: Round 1

• 39 senior and graduate level CS students were divided into two performance-balanced groups and given a quiz on a Java module containing 183 source lines of code.

• One group (the control) was provided the source in plain text only while the second group was provided the source with the CSD.

• Both groups were asked to respond to the same set of 12 questions.


Experimental Task

• All questions were related to the structure and execution of the code, but not to its overall functionality. For example:– How many ways are there to exit the loop that begins on line 91?– To what line would control be transferred immediately after

executing line 144?– How many conditions must be evaluated in order for line 152 to be

executed?

• Subjects were informed that they would be graded on both correctness and speed.– Answer as quickly as possible without sacrificing

correctness


Experimental Preparation

Performance balance of groups prior to experiment

0

2

4

6

8

10

12

14

A B C D F X

Course Grade Prior to Experiment

No

. o

f S

tud

en

ts

CSD

NoCSD


Experimental Results

• An initial analysis of differences in performance between the two groups was done using– average time to respond to each question (T1)– average time to respond correctly to each question

(T2)– number of correct responses across all questions

(T3)

• The data strongly rejected the null hypothesis that the CSD had no positive effect on subject performance in answering the 12 questions


Experimental ResultsTime taken to respond (T1)

020,00040,00060,00080,000100,000120,000140,000160,000180,000

1 3 5 7 9 11

Question

Tim

e (

mse

c)

CSD

NoCSD


Experimental ResultsTime taken to respond correctly (T2)

020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000

1 3 5 7 9 11

Question

Tim

e (

mil

lise

c)

CSD

NoCSD


Experimental ResultsNumber of correct responses (T3)

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10 11 12

Question

No

. S

tud

en

ts

CSD

NoCSD


Experimental ResultsTime taken to respond correctly (T2) - Revised

020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000

1 3 5 7 9 11

Question

Tim

e (

mil

lise

c)

CSD

NoCSD


Experimental Results

• Average time taken to respond (T1)– P-value = 0.0035

• Average time taken to respond correctly (T2)– P-value = 0.000305

• Number of correct responses (T3)– 45% of the CSD group’s responses were correct

while only 26% of the control group’s responses were correct

– Difference is highly significant (P-value = 0.0000167)


Follow-on experiment

CSD v. Plain Text: Round 2 • Experimental materials (code, questions) and

procedures were held constant while the subject population changed.– 50 CS 1 students

• Original results were partially reproduced: Statistically significant gains in accuracy and speed for the CSD group with respect to– average time to respond correctly (T2)– number of correct responses (T3)


Conclusions

• The CSD replaces penicillin as the greatest scientific advance of the 20th century.

• The IPO for GRASP, Inc. will make us fabulously wealthy.

• The beneficial effect of the CSD on program reading tasks is significant and measurable.

• There are still many more questions to answer…– Under what circumstances?– Do the benefits scale up to larger tasks?– Is there a pleasure or immediate reward factor that is

greater than the quantifiable effect?


Complexity Profile Graph

• Algorithmic level graph of complexity profile• Fine-grained metric

– for each production in the grammar

• Profile of program unit rather than single-value metric

• Complexity values from each measurable unit in a program are displayed as a set to form the complexity profile graph.

• Adds the advantages of visualization to complexity measurement.



• A program unit is parsed and divided into segments– e.g., each simple declaration or statement is a single

segment, composite statements are divided into multiple segments

• Each segment is a measurable unit.• Segments are non-overlapping and all code is

covered– i.e., all tokens are included in exactly one segment

• The complexity for each segment is a bar in the CPG.


Computing the CPG

• Content– C = ln(reserved words + operators + operands)

• Breadth– B = number of statements within a construct

• Reachability– R = 1 + number of operators in predicate path

• Inherent– I = assigned value based on type of control structure

• Total– T = s1C + s2B + s3R + s4I– where s1, s2, s3, s4 are scaling factors


CPG Study

Knapsack Program

0

2

4

6

8

10

12

14

q3 q4 q5 q6 q7 q8 q9 q10 q1 q2

Question

CP

G v

alu

e

Series1


CPG Study

Knapsack Program

0

10

20

30

40

50

60

q3 q4 q5 q6 q7 q8 q9 q10 q1 q2

Question

total(csd+ncsd)

Knapsack

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

q3 q4 q5 q6 q7 q8 q9 q10 q1 q2

Question

Err

or

Rat

e(%

)

Series1

Response Time Error Rate


Future Work

• Continue with the NSF-sponsored empirical research ...

• Other visualizations• Complexity (e.g., CPG)• Architecture (e.g., Object diagrams)

• Other reading techniques– source code folding

• Industrial case studies and experiments– Data collection framework

• jGRASP


GRASP Distributionhttp://www.eng.auburn.edu/grasp


Data Collection Framework• Allow the automated collection of high resolution utilization data from

GRASP sessions.• Allow the automated collection of various software measures.• We want to be able to capture how the user was interacting with the

tool and its visualizations at a given point in time as well as quantify the nature of the software with which they were working.

grasp – overview of program comprehension studies - slide 1 auburn university computer science and...

Documents

csd slide

end loop end loop end

requested loop

loop elsif response1

priority loop

nondisruptive slide

status repeat

null end