1

Asking and Answering Questions during a Programming Change Task,

By Jonathan Sillito, Member, IEEE, Gail C. Murphy,

Member, IEEE, and Kris De Volder.

Presented by Bob Mazzi

10/30/08

2

Introduction

What is the best approach to code changes? Not taught in school No industry standards No clear starting point

3

Introduction

Approach to this study What questions do programmers ask Can they be grouped into general categories How are questions answered Do existing tools support answering questions

to correctly make changes

4

Related Work

Program Comprehension Work involving the analysis of programmers’

Questions How programmers use tools and generally

how they carry out change tasks and other programming activities

5

Program Comprehension

Much work focuses on how programmers form a mental model of the code to be worked on

Top Down Model Brooks, Koenemann and Robertson, Soloway and Ehrlich

Bottom Up Model Pennington’s model Shneiderman and Mayer’s cognitive framework

Combined Models Littman et al. noted that programmers use either a “systematic” strategy or

an “as-needed” strategy Letovsky’s knowledge-based, opportunistic model Von Mayrhauser and Vans “integrated metamodel”

Tools to help programmer comprehension Von Mayrhauser and Vans - list of information needs and tool capabilities Walenstein’s theory of cognitive support – tools to support programmer

6

Work involving the analysis of programmers’ Questions

Johnson and Erdem & Herbsleb and Kuwana Studies of existing questions from usenet and actual design meetings

Letovsky & Erdos Kinds of questions - why, how, what, whether, and discrepancy

Sneed Types of questions

Ko et al. - Categorizing questions 1. writing code 2. submitting a change 3. triaging bugs 4. reproducing a failure 5. understanding execution behavior 6. reasoning about design 7. maintaining awareness

7

Studies of how programmers use tools and how they carry out change tasks and other programming activities

Storey et al User study focused on how program understanding tools

enhance or change the way that programmers understand programs

Flor and Hutchins Study on performing a change task

4 Recent studies on use of current development environments Robillard et al. how programmers navigate a code base Deline et al. report observing how programmers navigate Ko et al. research on IDE design elements for code

maintenance De Alwis and Murphy report on a field study about how

programmers experience disorientation

8

Case Study Design

First Study designed to use teams Promotes conversation Allows easy observation to develop list of questions

Second study used individuals ( and 1 team ) More common in industry

Study participants First study used students ( grad level ) Second study used experienced programmers

Code to be worked on First study used an unfamiliar Java application Second study used code familiar to programmers in a mix of languages

9

RESEARCH APPROACH

Study one Laboratory setting 9 participants Code that was new to programmers State-of-the-practice development tools Limited time designed to not allow completion Required changes span multiple files

Goals Easy to observe in order to gather questions

10

RESEARCH APPROACH

Study two Industrial setting 16 Participants ( 14 individuals, 1 team ) Familiar code Familiar tools and environment Most participants had 2 or more years experience

Goals Is the process different with familiar code Are different questions asked Focus on identifying problem and using tools

11

Analyzing basic study responses

Identification of questions Simplification / reduction of near duplicates

Categorization of questions Finding Focus Points Questions about a given entity and other entities directly related to it Understanding a number of entities and relationships together How groups relate to each other or to the rest of the system

Identification of tool use

An example of one programmers combination of tools from second study

12

Samples of categorizations and questions

Finding Focus Points 5 kinds of questions Which type represents this domain concept or this UI

element or action? Where in the code is the text in this error message or UI

element? Expanding Focus Points

15 kinds of questions Where is this method called or type referenced? When during the execution is this method called?

Understanding a Subgraph 13 kinds of questions How are instances of these types created and assembled? How are these types or objects related?

Questions over Groups of Subgraphs 11 kinds of questions How does the system behavior vary over these types or

cases? What are the differences between these files or types?

13

Question Frequencies

Some specific questions were more common in one study than the other Some questions were more likely to be asked by novices than

experienced coders Some questions would have been prior knowledge for the experienced

coders Some types of questions were more common in one study than the

other Advanced questions were more likely to be asked by experienced

coders Basic questions were more likely from coders unfamiliar with the code

being worked on Study parameter differences

As study one was designed as a team effort some questions may have been more likely to be stated to confirm that the partner agreed.

Results

14

Analysis of Tools Use for Answering Study Questions

Analyzing commercial and available research tools Comparing identified questions with available tools Categorizing tools by how they were able to support

answering specific questions No attempt was made to identify all possible tools,

just to identify if a tool was available that helped answer a specific question

15

Comparing existing tools to identified needs

Existing tools did well on basic questions

Existing tools are not well suited to answering some more complicated questions that were identified during this study

Tools do better at specific search type queries than less specific understanding type queries

TABLE 9The Number of Questions with

Full or Partial Support by Category

16

The gap between existing tools and identified needs

Support for More Refined or Precise Questions Scope of a search or query

Support for Maintaining Context Comparisons

Support for Piecing Information Together Information gathered from different sources is not

easily combined Multiple windows of related information add

confusion

17

Summary

These case studies have identified a list of common questions that are likely to be similar in other coding tasks

The comparison of tools capabilities to common questions has identified potential areas of improvement in tools

This paper has identified a need for tools that allow different questions and answers to be presented in a unified format

18

Limitations The use of pairs of programmers vs. individuals may have

introduced errors The comparison of Graduate students vs. Experienced

programmers may have introduced errors The use of familiar and unfamiliar code in the two studies may

have introduced errors The designed time limitation may have created a limitation in

which questions that might be asked later in the process never appeared in the limited time available

These studies represent a small sample size as each study was a combination of several smaller tasks. This implies that each task was either not repeated at all or in some cases only a few times and the repeats were all among the novices

These studies used a limited set of tools. A larger or different set of tools may produce different results

19

Comments – Future Work

Each limitation listed also may be regarded as adding breadth to the studies as they introduce additional questions and potential combinations

The researchers recorded gender for each study participant yet did not explore the potential differences in questions asked, tool use or logical approach implemented by each gender

20

Discussion

21

Finding Focus Points1. Which type represents this domain concept or this UI element or action? 2. Where in the code is the text in this error message or UI element? 3. Where is there any code involved in the implementation of this behavior? 4. Is there a precedent or exemplar for this? 5. Is there an entity named something like this in that unit (project, package,

or class, say)?

22

Expanding Focus Points6. What are the parts of this type? 7. Which types is this type a part of? 8. Where does this type fit in the type hierarchy? 9. Does this type have any siblings in the type hierarchy? 10. Where is this field declared in the type hierarchy? 11. Who implements this interface12. Where is this method called or type referenced? 13. When during the execution is this method called?14. Where are instances of this class created? 15. Where is this variable or data structure being accessed?16. What data can we access from this object? 17. What does the declaration or definition of this look like? 18. What are the arguments to this function?19. What are the values of these arguments at runtime?20. What data is being modified in this code?

23

Understanding a Subgraph21. How are instances of these types created and assembled? 22. How are these types or objects related? (whole-part)23. How is this feature or concern (object ownership, UI control, etc.)

implemented? 24. What in this structure distinguishes these cases?25. What is the behavior that these types provide together and how is it

distributed over the types?26. What is the “correct” way to use or access this data structure? 27. How does this data structure look at runtime? 28. How can data be passed to (or accessed at) this point in the code?29. How is control getting (from here to) here?30. Why is not control reaching this point in the code? 31. Which execution path is being taken in this case? 32. Under what circumstances is this method called or exception thrown? 33. What parts of this data structure are accessed in this code?

24

Questions over Groups of Subgraphs

34. How does the system behavior vary over these types or cases? 35. What are the differences between these files or types? 36. What is the difference between these similar parts of the code (e.g.,

between sets of methods)? 37. What is the mapping between these UI types and these model types?

25

26