Application Domain Knowledge and Programmers’ Mental Representations

Teresa M. ShaftUniversity of Oklahoma

Iris VesseyIndiana University

Application Domain Knowledge

Application Domain refers to the context of the problem to be addressed by the computer software – in contrast to the solution or computing domain (Blum 1989)

adapted from Blum, 1989

Implementation

Domain

Application

Domain

Conceptual

Models

Formal

Models

The Software Development Process

Importance of Application Domain Knowledge in Software Development

Those who can map deep application domain knowledge into a computation solution often serve as project gurus

“the thin spread of application domain knowledge” was one of the most salient problems in large scale development Curtis, Krasner & Iscoe 1988

Significant effort during software development is associated with acquiring needed application domain knowledge Walz, Elam & Curtis 1993

Application Domain Knowledge Acknowledged as critical to the ability to

develop, comprehend, and maintain software (Brooks, 1990, Pennington & Grabowski 1990, Vans, von Mayrhauser & Somlo 1999)

Infrequently studied (Glass & Vessey 1998).

Most studies of software comprehension or maintenance either do not consider the application domain or use materials the researchers argue do not require application domain knowledge.

Application Domain Knowledge in Software Comprehension

Programmers use a more top-down (hypothesis driven) comprehension process when they have relevant application domain knowledge (Shaft & Vessey 1995)

During software comprehension, programmers use application domain knowledge in addition to (rather than to replace) programming knowledge (Shaft & Vessey 1998)

Current Study

Examines the influence of application domain knowledge on programmers’ mental representations during the software comprehension and enhancement process

Examines the influence of different types of enhancement tasks on the development of programmers’ mental representations

Comprehension Process

Program Code and

Documentation

Mental Representation

of Computer Program

Enhancement Specification

Enhancement Process

Mental Representation

of Enhanced Program

Enhanced Computer Program

Application Domain

Knowledge

Programming Domain

Knowledge

Programmer’s Knowledge Base

Program/ProgrammerSpecific Knowledge Base

Program/ProgrammerSpecific Knowledge Base

Conceptual Model of the Software Comprehension and Enhancement

Programmers’ Mental Representations

Internal knowledge structure of the contents of the computer program

Numerous types of information are embedded in a computer program (Brooks 1987, Pennington 1987, Green 1977, Corritore & Wiedenbeck 1999)

Comprehension is the process of extracting that information from the computer program

Programmers’ mental representations reflect their comprehension of the different types of information embedded in the programs

Knowledge Categories

Function Data flow Control flow State

Pennington 1987a, 1987b These categories are relevant to

procedural programs

Initial Comprehension of a Computer Program

Driven by the knowledge programmers’ bring to the comprehension task

Hypothesis 1: Initial Comprehension

H1a: Programmers have higher levels of initial comprehension when they are familiar with the application domain of the program than when they are unfamiliar with the application domain.

H1b: When programmers are familiar with the application domain, they have better comprehension of function and data flow information than of control flow and state information.

Comprehension After Conducting an Enhancement

Influenced by the nature of the enhancement task as well as the knowledge programmers’ possess

Different types of enhancement tasks rarely considered

Consider two types of enhancement tasks: Function – high need for application

domain knowledge Control flow – low need for application

domain knowledge

Hypotheses 2: Comprehension After Conducting An Enhancement

H2a: Comprehension after conducting an enhancement will be greater when programmers are familiar with the application domain (than when they are unfamiliar with the application domain).

H2b: Programmers have greater comprehension of the type of knowledge emphasized in the enhancement task than the other types of knowledge. Control flow task -> control flow knowledge Function task -> function knowledge

Changes in Programmers’ Comprehension

Influenced by programmers’ knowledge

Influenced by the nature of the enhancement task

Hypotheses 3: Changes in Comprehension

H3a: Programmers experience greater changes in comprehension when they are familiar with the application domain (than when they are unfamiliar)

H3b: Programmers who conduct a control flow task have greater increases in comprehension than those who conduct a function task

Methodology 24 professionals studied and

enhanced two computer programs Minimum of 2 years professional

experience Computer programs from two

application domains Familiar application domain: accounting Unfamiliar application domain: hydrology COBOL programs Equivalent size, data density, decision

density

Methodology (continued) Assessed programmers’

comprehension at two points: After an initial study period After conducting an enhancement task

Comprehension assessed via questions Two questionnaires per program 20 questions per questionnaire 5 questions for each knowledge

category

Methodology (continued)

Half of the programmers conducted control flow enhancements on both programs; half conducted function enhancements

Control flow tasks: insert a new level of control break in an existing control break report

Function tasks: create a new capability Enhanced programs of equivalent size,

data density & decision density

Analysis of Hypothesis 1

Source DF SS Mean Square F Value Pr > F

Application Domain (AD) 1 14700.00 14700.00 16.74 .001Error 23 20200.00 878.26

KnowledgeCategory (KC) 3 8625.00 2875.00 7.61 .01Error 69 26075.00 377.90

AD * KC 3 466.67 155.56 .44 .73Error 69 24633.33 357.00

Hypothesis 1 – Initial Comprehension

H1a: supported programmers’ have higher levels of

comprehension in the familiar application domain

H1b: not supported No interaction between knowledge category

and application domain familiarity Programmers’ comprehension of state

knowledge was significantly greater than data flow, regardless of knowledge of the application domain

Initial Comprehension by Knowledge Category

70.84

59.17

69.17

73.33

53.33

38.33

50.00

30

40

50

60

70

80

90

100

Function Data Flow Control Flow State

Knowledge Category

Percent Correct

Familiar(Accounting)Unfamiliar(Hydrology)

Analysis of Hypothesis 2Source DF SS Mean Square F Value Pr > F

Type of EnhancementTask (TET) 1 1752.08 1752.08 1.44 .24Error 22 26845.83 1220.27

Application Domain (AD) 1 10502.08 10502.08 31.21 <.001AD * TET 1 4680.75 468.75 1.40 .25Error 22 7389.58 335.42

KnowledgeCategory (KC) 3 7289.58 2429.86 10.81 <.001KC * TET 3 522.92 174.31 .78 .51Error 66 14837.50 224.81

AD * KC 3 5972.92 1990.97 6.65 .001AD * KC * TET 3 2106.25 702.08 2.34 .08Error 66 19770.83 299.56

Hypothesis 2 – Comprehension After Conducting the Enhancement

H2a: supported Comprehension in the familiar application domain

is greater than in the unfamiliar application domain H2b: not supported

Type of enhancement task did not influence which knowledge categories are better comprehended

Application domain knowledge influenced comprehension of the different knowledge categories: Data flow, control flow and state knowledge more

accurately understood in the familiar application domain

Within familiar application domain: state knowledge more accurately understood than other categories

Within unfamiliar application domain: state and function more accurately understood than data flow knowledge

Application Domain Familiarity by Knowledge Category After Enhancement Task

86.67

72.50 71.67

66.67 65.00

57.50

45.83

70.00

30

40

50

60

70

80

90

100

Function Data Flow Control Flow State

Knowledge Category

Percent Correct Familiar (Accounting)

Unfamiliar (Hydrology)

Application Domain:

Analysis of Hypothesis 3Source DF SS Mean Square F Value Pr > FType of Enhancement Task (TET) 1 176.04 176.04 0.06 .81Phase 1 5551.04 5551.04 16.14 <.001TET * Phase 1 2109.38 2109.38 6.13 .02Error 22 7564.58 343.84Application Domain (AD) 1 25026.04 25026.04 38.85 <.001AD * TET 1 26.04 26.04 0.04 .84Error 22 14172.92 644.22 Knowledge Category (KC) 3 15586.46 5195.49 15.51 <.001KC * Task 3 786.46 262.15 0.78 .51Error 66 22102.08 334.88Phase * AD 1 176.04 176.04 0.29 .59Phase*AD*TET 1 651.04 651.04 1.09 .31Error 22 13197.92 599.91 Phase*KC 3 328.13 109.38 .41 .75Phase*KC*TET 3 969.79 323.26 1.21 .31Error 66 17577.08 266.32ADK * KC 3 3253.13 1084.38 4.05 .01ADK* KC* TET 3 1553.13 517.71 1.93 .13Error 66 17668.75 267.31Phase*ADK*KC 3 3186.46 1062.15 2.65 .06Phase*ADK*KC*TET 3 844.79 281.60 .71 .55Error 66 26443.75 400.66

Hypothesis 3 – Changes in Comprehension H3a: not supported, application domain

knowledge did not allow programmers to gain more knowledge

H3b: supported, those who conducted a control flow enhancement showed greater gains in comprehension

Application domain influenced the types of knowledge programmers understood: Higher levels of comprehension of all knowledge

categories in the familiar application domain Within familiar application domain: state

knowledge more accurately understood than function and data flow

Within unfamiliar application domain: function, control flow and state knowledge more accurately understood than data flow

Application Domain Familiarity by Knowledge Category Interaction Across Phases

80

68.75

65.42

70.83

61.67

42.08

53.75

62.92

30

40

50

60

70

80

90

Function Data Flow Control Flow State

Knowledge Category

Percent Correct Familiar (Accounting)

Unfamiliar (Hydrology)

Application Domain:

Phase by Type of Enhancement Task Interaction

70 Control Flow

57.71

63.96 Function 61.04

30

40

50

60

70

80

Prior to Enhancement After Enhancement Task

Phase

Percent Correct

Control Flow

Function

Type of Enhancement Task:

Summary of ResultsResearch Question 1: Initial

Comprehension2: Comprehension After

Conducting the Enhancement 3: Changes in Comprehension

Application Domain

Familiarity

Familiar > unfamiliar

Familiar > unfamiliar. Application domain knowledge did not result in greater gains in

comprehension.

Type of Knowledge

State > data flow

Across Application Domains: Data flow, control flow & state knowledge better understood in the familiar than unfamiliar

application domain.Within Familiar Application

Domain: state > function, data flow & control flow.

Within Unfamiliar Application Domain: function & state >

data flow.

Across Applications Domains: all knowledge categories better

understood in the familiar application domain.

Within Familiar Application Domain: state > function & data

flow. Within Unfamiliar Application Domain: function, control flow

& state > data flow.

Type of Enhancement

Task

Not applicable Type of enhancement task did not influence programmers’

comprehension or their understanding of different

types of knowledge

Conducting a control flow enhancement increased overall

comprehension.

Discussion

When programmers possess application domain knowledge they have higher levels of comprehension initially and after enhancing a program Consistent for two types of enhancements

Even in the context of program maintenance, a software development task that is seen as relatively far from the original domain, relevant application domain knowledge benefits software comprehension

Discussion

Conducting a control flow enhancement task led to gains in comprehension Not all types of tasks will give

programmers’ the same opportunity to develop their mental representation of a computer program

A more complete taxonomy of tasks would be helpful to future researchers

Implications – Types of Enhancement Tasks

Long-term vs. short-term employees Programmers’ mental representations

develop if they engage in tasks that require changes to multiple locations of the program

Programmers who conduct enhancements which do not require interacting with much of the original program, did not experience significant increases in comprehension

Implications – Application Domain Knowledge

Lengthy start-up time for new IT employees

Training and selection Outsourcing and off-shoring

Activities that require less application domain knowledge would seem to be better choices

Develop mechanisms to provide application domain knowledge

Conclusions Not all enhancement tasks encourage

development of programmers’ mental representation of a computer program.

Traditionally, software comprehension and enhancement were seen as residing in the programming domain and requiring little, if any, application domain knowledge.

Application domain knowledge is beneficial to software comprehension, and to the understanding of all knowledge categories.