perception-action icons: an interface design strategy for...

INTRODUCTION

Cognitive systems engineering (Rasmussen,Pejtersen, & Goodstein, 1994) provides a generalframework for the development of effective com-puterized decision support. The foundation of thisapproach is that an analysis and description of do-main constraints (i.e., the regularity in a domainor, alternatively, the nature of the work to be done)is essential in developing effective interfaces. Ras-mussen et al. (1994) have developed a continuumto categorize different domains in terms of theirconstraints. At one end of the continuum are do-mains in which the unfolding events arise from thephysical structure and functionality of the systemitself (e.g., process control). In these “law-driven”

domains, highly trained and frequent users respondto demands that are created by the domain. At theopposite end of the continuum are “intent-driven”domains in which the unfolding events arise fromthe user’s intentions, goals, and needs (e.g., infor-mation search and retrieval). Users typically inter-act with these systems on a more casual basis, andtheir skills, training, and knowledge are far moreheterogeneous.

The interface design strategy that will be suc-cessful for a particular domain is determined by thedomain’s location on this continuum. The cogni-tive systems engineering literature has provided ex-cellent examples of design strategies for domainsthat fall at either of the two ends of the continuum.The most effective design strategy for law-driven

Perception-Action Icons: An Interface Design Strategy forIntermediate Domains

Christopher P. Talcott, U.S. Military Academy, West Point, New York, Kevin B. Bennett,Wright State University, Dayton, Ohio, Silas G. Martinez and Lawrence G. Shattuck, U.S.Military Academy, West Point, New York, and Craig Stansifer, Wright State University,Dayton, Ohio

Objective: A prototype interface was developed to support decision making duringtactical operations; a laboratory experiment was conducted to evaluate the capabili-ty of this interface to support a critical activity (i.e., obtaining the status of friendlycombat resources). Background: Effective interface design strategies have beendeveloped for domains that have primarily law-driven (e.g., process control) or intent-driven (e.g., information retrieval) constraints. However, design strategies for inter-mediate domains in which both types of constraints are equally critical, such as militarycommand and control, have not been explored as extensively. The principles of directperception, direct manipulation, and perception-action loops were used to develop ahybrid interface design strategy (“perception-action icons”) that was incorporated intothe prototype interface. Methods:Aqualitative tactical simulation and an alternativeinterface (an experimental version of an existing U.S. Army interface) were developed.Participants used both interfaces to provide estimates of friendly combat resourcesfor three different categories of information at three different echelon levels. Results:The results were unequivocal, indicating that the interface with perception-action iconsproduced significantly better performance. Conclusion: The perception-action icondesign strategy was very effective in this experimental context. The potential for this de-sign strategy to be useful for other intermediate domains is explored. Application:Actual or potential applications of this research include both specific interface designstrategies for military command and control and general interface design principlesfor intermediate work domains.

Address correspondence to Kevin B. Bennett, Department of Psychology, 335 Fawcett Hall, Wright State University, Dayton,OH 45435; [email protected]. HUMAN FACTORS, Vol. 49, No. 1, February 2007, pp. 120–135. Copyright © 2007,Human Factors and Ergonomics Society. All rights reserved.

PERCEPTION-ACTION ICONS 121

domains is to develop analogical visual displaysthat utilize geometrical forms to directly reflect do-main constraints (e.g., Vicente, 1991). The mosteffective design strategy for intent-driven domainsis to develop spatial metaphors (e.g., the desktopmetaphor) that relate interaction requirements tomore familiar concepts and activities (e.g., Pej-tersen, 1992).

The design strategy (or perhaps strategies) thatis appropriate for domains that fall in the middleof this continuum is less clear. These domains arecharacterized by the presence of both law-drivenconstraints and intent-driven constraints that areroughly equivalent in terms of their importancein shaping overall system behavior. The term in-termediate will be used to describe this generalcategory of domains. Agood example of an inter-mediate domain is military command and control.There are law-driven constraints that arise from anextensive technological core (weaponry, sensors,communication, etc.). However, there are alsointent-driven constraints. The difference in inten-tions between friendly and enemy forces is one ob-vious factor, but intent also plays a substantial rolewithin a military organization. For example, dur-ing tactical engagements lower-level leaders basetheir actions upon an interpretation of the com-mander’s intent statement in mission orders (e.g.,Klein,1994). One solution to the design challengespresented by this category of domains was imple-mented in a prototype interface for U.S. Armycommanders. This solution will be described in thecontext of the principles of cognitive systems en-gineering and ecological interface design thatguided its development.

Perception-Action Icons for MilitaryCommand and Control

Gibson’s (1966) theoretical work in experimen-tal psychology has provided valuable insights forinterface design. The simple example of naviga-tion through the natural environment will be usedto illustrate some of these concepts. The ambientoptical array is specified by (and corresponds to)the structure of the natural environment. The ac-tions performed by an agent (i.e., reorientation ofthe eyes, turning of the head, ambulation) producesystematic transformations of the optical array thatare referred to as optical invariants (e.g., opticalflow). The brain “resonates” to this information;the information is “obtained” or “picked up.” This,

in turn, provides affordances or possibilities foraction by the agent. The agent coordinates and ad-justs his or her actions based on feedback obtainedfrom the continuous space-time signals that arisefrom (and are specific to) the combination of struc-ture in the environment and observer action. Thus,successful interaction with the natural environ-ment depends upon a dynamic and continuousperception-action loop that draws upon highlyefficient skill-based behaviors (Rasmussen, 1983).

The implication for interface design is that thedisplays and controls in an interface should bedesigned to maintain an intact perception-actionloop, thereby facilitating interaction. The Repre-sentation Aiding Portrayal of Tactical OperationsResources (RAPTOR) interface achieves this goalthrough “perception-action” icons that provideintegrated display (direct perception) and control(direct manipulation) design components that pre-serve the loop’s integrity. Each component will bedescribed in greater detail.

Direct perception. In describing direct percep-tion, Rasmussen et al. (1994) observed, “In Gib-son’s terms, the designer must create a virtualecology, which maps the relational invariants ofthe work system onto the interface in such a waythat the user can read the relevant affordances foractions” (p. 129). The abstraction (means-ends)and aggregation (part-whole) hierarchies are ana-lytical tools that have been developed to discoverthe constraints (i.e., the relational invariants) of awork domain. An analysis of U.S. Army tacticaloperations at the battalion level was conductedusing these tools (Martinez, Bennett, Talcott,Stansifer, & Shattuck, 2001); a partial listing isprovided in the left section of Table 1. Achievingdirect perception requires at least two differentsets of mappings. One set of mappings involvesthe relationship between the constraints of thework domain and the informational content en-coded into the graphical representations (i.e., areappropriate categories of domain informationand relations available in the interface?). This willbe referred to as content mapping.Asecond set ofmappings involves the relationship between thevisual properties of the graphical representationsand the perceptual capabilities and limitations ofthe observer (i.e., have the domain constraints beenencoded or represented in the interface so that theycan be easily obtained?). This will be referred toas form mapping. The quality of these mappingswill determine the extent to which the affordances

122 February 2007 – Human Factors

of the domain, and therefore the potential forappropriate control actions to be executed, will beavailable for pickup by the observer.

The first set of mappings (i.e., content) for theRAPTOR interface is summarized in the right-hand side of Table 1; an example of the graphicalformat (a unit at the battalion echelon level) andassociated “roll-over” behaviors is illustrated inFigure 1. The combat power of a unit (i.e., its mil-itary force) is a fluctuating commodity that ebbsand flows according to resources expended duringbattle and resources gained during reinforcement.The categorical status of a unit’s combat power isrepresented by the background color code (i.e.,green: 100%–85%; amber: 84%–70%; red: 69%–50%; and black: <49%) of the unit’s icon. This in-formation resides at the level of abstract functionsand priority measures. The tangible contributionsto a unit’s combat power are determined by thevalues of five combat parameters (tanks, Bradleypersonnel carriers, ammunition, fuel, and person-nel). The percentage of each parameter is repre-sented by the vertical position of an analog markerrelative to the bottom (0%) and top (100%) of thecombat parameter mats and by digital values (seeFigure 1); the categorical status of each parameteris represented by the color of this mat. This infor-mation corresponds to the level of physical proc-esses and activities in the abstraction hierarchy.

Other information at this level includes therange arc specifying the weapons envelope (Fig-ure 1b) for the unit’s primary munition and the

unit’s identification, type, and size symbols. Theunit’s role in the current tactical operation is indi-cated through standard U.S. Army activity symbols(i.e., the graphic marked as “unit activity symbol”in Figure 1b). This information corresponds to thelevel of general functions and activities. Informa-tion at the level of physical form and configurationincludes the physical characteristics of the battle-field terrain and the physical location of the uniton this terrain.

The second set of mappings (i.e., form) in-volves the relationship between the visual prop-erties of the display and the observer’s perceptualcapabilities and limitations. In tactical operationsthe combat power of a unit is perhaps the most crit-ical information to be presented; the tangible con-tributions to a unit’s combat power consist of thefive combat parameters. At least two categories ofgraphical formats could be used: (a) a display inwhich the combat parameters are mapped into asingle geometrical form and (b) a display in whicheach combat parameter has its own unique rep-resentation. Both of these formats can produceemergent features and therefore can qualify as“configural” displays; however, to facilitate dis-cussion the former will be referred to as config-ural displays and the latter will be referred to asseparable displays.

The proper choice between these two formatsdepends upon the inherent relationships betweenthe domain variables to be presented (Bennett &Flach,1992). Aconfigural display is the appropriate

TABLE 1: Abstraction Hierarchy Analysis of Army Battalion During Tactical Operations and CorrespondingVisual Indicators in RAPTOR Interface

Abstraction Hierarchy Military Tactical Operations RAPTOR Icon

Goals, purposes, and Mission objectives, collateral damage,constraints public perception, etc.

Abstract functions and Combat power, value of mission objectives Categorical combatpriority measures vs. resource expenditure, probability of power indicator for unit

success/failure, etc.

General functions and Command and control, maneuver, combat Activity symbol activities service support, air defense, intelligence, indicating role in battle

fire support, mobility and survivability, etc.

Physical processes and Vehicles (speed, maneuverability), Quantitative indicators activities weapons (power, range), sensors for combat parameters;

(sensitivity, range), terrain (avenues range fan for primary of approach), etc. munition

Physical form and Physical location of units, physical Terrain map, position ofconfiguration characteristics of terrain and weather, etc. graphical form


Primary Munitions Range Envelope Unit Activity Symbol

Digital Combat Parameter Values

Unit Identification Number

Unit Size Symbol

Unit Type Symbol

Task Force Icon

Combat Parameter Mats

Elapsed Time Indicator

Alphanumeric Combat Parameter Categorical Status Indicators

Analog Combat Parameter Percentage Indicator

A.

B.

Figure 1. Direct perception in the RAPTOR interface. (a) Key elements of the perception-action icons in the RAPTORinterface (illustrated at the battalion level). (b) Additional “rollover” information that appears when the mouse ispositioned over a unit icon (illustrated at the company level).


choice when the individual variables are tightlycoupled (i.e., interactions between individual vari-ables produce higher order domain properties).Under these circumstances a properly designedconfigural display will produce salient, higher lev-el visual properties (i.e., emergent features) thatcorrespond to these higher order domain proper-ties. However, when the individual variables areloosely coupled (they do not necessarily interact toproduce well-defined higher level domain prop-erties), a configural display will produce salientvisual properties (i.e., emergent features) that aremeaningless yet at the same time quite difficult toignore. See Bennett and Flach (1992) and Bennettand Fritz (2005) for more detailed discussions ofthese and related issues.

The domain analyses revealed that the fivecombat parameters are not tightly coupled: the re-lationship between them can vary substantiallyacross different contexts (e.g., offensive vs. defen-sive missions). Thus, the proper design choice forthe RAPTOR interface was to incorporate uniquerepresentations for each of the combat parameters(see Figure 1). Note that the pattern of relativeheights for the analog indicators can produce alimited form of configurality (Sanderson, Flach,Buttigieg, & Casey, 1989) that matches the loose-ly coupled relationships that characterize the com-bat parameters.

A second set of considerations in form map-ping involves more specific characteristics of thedisplay. The constraints in complex, dynamic do-mains will be hierarchically structured and nested;there is a corresponding need to visually segregatethe information that appears in the interface (seeTufte, 1990). The challenge is to provide visual in-formation that reflects the inherent structure andthe relative importance of the corresponding do-main information. Effective mappings at this levelwere devised for the RAPTOR interface. For ex-ample, the most critical piece of information (com-bat power of a unit) is represented by the mostsalient visual feature in the graphical format (thebackground color code of the unit’s icon). Infor-mation at an intermediate level of importance (in-dividual combat parameters, their values, and theirrelationships) was presented through visual features(background parameter mats, analog percentageindicators) at an intermediate level of salience.Basic information (e.g., unit’s identification, type,and size symbols) was presented at the lowest lev-el of visual salience. Finally, some information

(munition envelope, activity symbol, digital valuesof combat parameters) was available only whenthe mouse rolled over an icon, thereby providingaccess to this information but avoiding clutter.

Direct manipulation. The concept of directmanipulation has a long history in the human-computer interaction literature. For example, morethan 20 years ago Hutchins, Hollan, and Norman(1986) provided an extensive analysis. It has sincebecome a platitude that the interface should bedesigned so that objects can be manipulated direct-ly. In reality, this goal is rarely achieved. For ex-ample, pull-down menus have become universalin today’s application software. Although they area clear improvement over command line inter-faces, they do not constitute direct manipulation:The objects of interest are really not being manip-ulated directly. Recently, Burns and Hajdukiewicz(2004) emphasized this point by referring to directmanipulation – “where users feel as if they areworking directly with the object and not with theinterface” – as “the ‘holy grail’ of interface de-sign” (p. 2).

Adescription of decision-making requirementsduring tactical operations is needed to appreciatehow the icons in the RAPTOR interface were de-signed to achieve the design goal of direct manip-ulation. Units at each echelon level can pursuecollective or individual mission objectives. There-fore, an essential requirement is to consider thecombat resources and activities at each echelonlevel. This imposes substantial demands on thecommander and his or her staff. There are at least17 echelon levels that a commander needs to con-sider: 1 at the battalion level, 4 at the companylevel (A, B, C, and D) and 12 at the platoon level(1, 2, and 3 for each company). To complicate mat-ters further, the commander is required to trackthe five combat resources for each of these unitsand may need to consider these resources in thecontext of the battlefield terrain, given that it hasa substantial impact on a variety of factors rele-vant to tactical operations.

Direct manipulation of the graphical icons (i.e.,pointing at and clicking on them) in the RAPTORinterfaces allows an individual to execute criticalcontrol functions regarding echelon level. The de-fault configuration of the RAPTOR interface pre-sents the four company icons on the contour mapand the battalion icon off the map (Figure 2a).Clicking an icon on the contour map provided afiner grain of resolution. For example, a click on


A.

B. C.

Figure 2. Direct manipulation in the RAPTOR interface. (a) Default configuration of RAPTOR with company iconslocated on the contour map and battalion icon located in the holding area. (b) Manipulation of an icon on the contourmap produces view of combat resources at a finer grain of resolution. (c) Manipulation of an icon in the holding areaproduces a view of combat resources at a coarser grain of resolution.


the D company icon circled in Figure 2a wouldmove this icon off the map to the location indicat-ed by the left arrow and place the three associatedplatoon icons on the contour map (Figure 2b).Conversely, clicking an icon located off the con-tour map, such as the battalion icon in Figure 2a,would move it onto the contour map as indicatedby the right arrow and place the four associatedcompany icons off the contour map (Figure 2c).Thus, the objects of interest (icons representingunits of action) are manipulated directly to changethe resolution and the context in which informationabout friendly combat resources was displayed.

The instantiation of the perception-action iconsdesign strategy in RAPTOR produced an interfacethat stands in sharp contrast to an existing, and par-tially fielded, U.S. Army interface: the Force XXIBattle Command Brigade and Below (FBCB2) in-terface. An experimental version of this interfacewas developed, and the features that are relevantfor obtaining friendly forces information will bedescribed. Figure 3 illustrates the main screen ofthe FBCB2 interface; symbols for the four com-panies of a battalion are present on the contourmap (Figure 3a). The “F3 Combat Msgs” button isclicked to access company-level information re-garding combat resources. The “combat messages”screen (Figure 3b) appears and presents a matrixwith rows corresponding to the four companies(e.g., “D/3-66”) and columns corresponding tocombat readiness parameters (e.g., “fuel”). Thematrix cells present a categorical estimate of eachcompany’s combat parameter strength through col-or coding and a letter indicator (e.g., “B” for black).More detailed data can be obtained by activatingthe company’s button in the left-most “Unit” col-umn (e.g., “D/3-66”), which produces a “long formmessage” screen containing an alphanumeric datasheet (see Figure 4a). The “FIPR 12” (Figure 3a,top right corner) button is clicked to access platoon-level information. The “FIPR” screen (Figure 4b,“Flash-0” tab activated) then provides access toa military “E-mail inbox” with platoon informa-tion (e.g., “1/A/3-66AR”). Clicking the buttonsin the “Time,” “Msg Type,” or “Source Origina-tor” columns provides a detailed alphanumericdata sheet similar to that in Figure 4a.

Field studies of the FBCB2 interface (Centerfor Army Lessons Learned, 1997; Prevou, 1995)indicate that commanders and their staffs were in-undated by the amount of data presented and theamount of effort required to interpret these data,

particularly during combat situations when highstress and heavy workloads were imposed. Acon-crete example of these difficulties is illustrated bythe activity sequence required to obtain the valueof a combat resource (e.g., number of tanks) at thebattalion level. The “F3 Combat Msgs” button isactivated from the main screen (Figure 3a). Acom-pany button (e.g., “D/3-66”) is then activated inthe combat messages screen (Figure 3b), and thecorresponding long form message screen appears(Figure 4a). The parameter value must be locatedin the alphanumeric data and remembered. The en-tire process must be repeated for three more com-panies, followed by the computation of the finalparameter value (either mentally or manually).

The RAPTOR interface, designed specificallyto support direct perception and direct manipula-tion, provides far better interface resources for thecompletion of this task. The user simply needs toactivate (i.e., click on) the battalion icon and mouseover it (after it appears on the contour map) to viewthe computed parameter value (assuming the de-fault configuration of the RAPTOR interface illus-trated in Figure 2a). Although space limitations donot permit a thorough description, similar advan-tages are present with RAPTOR for different in-formational needs (e.g., the categorical status oftanks) and different echelon levels.

The previous discussion suggests that theRAPTOR interface will be more effective than theFBCB2 interface in obtaining friendly forces in-formation. This hypothesis was tested in a con-trolled laboratory setting. Aqualitative simulationwas developed to portray realistic changes in com-bat resources at three time periods during an offen-sive tactical scenario. Active U.S. Army officersserved as participants and were required to per-formed well-constrained but critical tasks. Threetypes of assessments were administered with re-gard to the combat readiness of friendly forces:quantitative (e.g., “What is the numerical value oftanks in Company C?”), categorical (e.g., “Whatis the color-code status of fuel in the Battalion?”),and needs (e.g., “What platoon in Company Bneeds Bradleys?”). Participants were also requiredto consider friendly forces at three different eche-lon levels (battalion, company, platoon). These as-sessments are typical of the information-seekingactivities that U.S. Army commanders would per-form repeatedly during the course of tactical oper-ations. It was predicted that the RAPTOR interfacewould improve performance.


A.

B.

Figure 3. FBCB2 Interface 1. (a) Default configuration of FBCB2 interface (main screen). (b) Activating the “F3”button in default configuration (Figure 3a) produces a pop-up window with a categorical summary of combatresources at the company level (“Combat Messages”).


A.

B.

Figure 4. FBCB2 Interface 2. (a) Activating the company button (e.g., “D/1-22”) in the combat messages screen(Figure 3b) produces pop-up window with alphanumeric descriptions of combat resources at the company level(“long form message”). (b) Activating the “FIPR 12” button in default configuration (Figure 3a) produces a pop-upwindow with a list of buttons corresponding to platoons (and long form messages upon their activation).


METHOD

Participants

Twelve male U.S. Army officers (6 captains, 6sergeants first class) volunteered to participate.Their military specialties were engineer, artillery,or armor (8–20 years of active-duty service), andthey ranged from 30 to 41 years of age. No par-ticipants had previous experience with eitherinterface. All participants had normal or corrected-normal visual acuity and color perception.

Apparatus

All experimental events were controlled byidentical computers (Apple Computer, Inc., Cu-pertino, CA, G3-300 MHz), with identical colormonitors (Dell Computer, Round Rock, TX, Trin-itron, 40.64 cm, 1024 × 768 resolution, ModelD1025TM) and standard keyboards. Participantswere also provided with notepaper, a pen, and acalculator.

Simulation Model

Asimulated offensive tactical scenario was de-veloped, based on exercises conducted at the U.S.Army’s National Training Center. The battalion’smission was to traverse a predefined route, engagethe enemy, defeat the enemy, establish a defensiveposition, and prepare for a counterattack. Therewere four companies in the battalion: CompaniesA (10 tanks and 4 Bradleys), B (14 tanks), C (4tanks and14 Bradleys), and D (14 Bradleys). Therewere three platoons (Platoons 1, 2, and 3) in eachcompany (each with 4 tactical vehicles – either alltanks or all Bradleys). The combat resources foreach echelon level (battalion, company, platoon)were considered at three different points in time:H-hour (onset of initial engagement), H + 3 (3 hr later), and H + 12 (12 hr later). The combatresources consisted of five combat readiness para-meters: tanks, Bradleys, ammunition, fuel, and per-sonnel. Three of these parameters (tanks, Bradleys,personnel) were computed as a simple percentageof the full complement. Ammunition was com-puted as the number of potential armored vehiclekills (all 120-mm rounds + all antitank missilerounds + the 25-mm rounds/10). Fuel was com-puted as the unit range in kilometers (using the fueleconomy of the M1 Tank).

Interfaces

The Introduction section provides details re-

garding the two interfaces, and only minor quali-fications will be provided here. Two of the authors(Talcott, Martinez) completed an abbreviated U.S.Army course on the actual FBCB2 interface. Theexperimental FBCB2 interface was designed toreplicate the visual appearance and selected func-tionality of this interface as it appeared in Decem-ber 2000. The experimental interface differed fromthe actual interface in three respects. First, tankand Bradley resources were separated. Second,fuel and ammunition were calculated as range andpotential kills (instead of gallons and rounds, asdescribed previously). Third, platoon-level datawere simplified (only platoon status E-mail mes-sages appeared) and more organized (listed in or-der of company or platoon, not in the order theywere received). These changes were introduced toprovide equivalent information, thereby makingcomparisons with the RAPTOR interface moremeaningful.

Procedure

Participants completed four sessions on suc-cessive days. In the training session (2 hr) all parti-cipants received both written and oral descriptionsof the simulation, interfaces, and experimentaltasks and completed a practice session using bothinterfaces. Participants then completed one exper-imental session (1hr) on each of 3 successive days.Each experimental session contained six blocksof trials formed by a factorial combination of thetwo interfaces and the three combat phases. Theorder of these six blocks was randomized.

Three types of questions were administered. A“quantitative assessment” question asked for thequantitative value of a combat readiness parame-ter (tanks, Bradleys, ammo, fuel, personnel) for aparticular unit (e.g., “What is the numerical valueof tanks in Company C?”). A“categorical assess-ment” question asked for the categorical code(e.g., black, red, amber, or green) of a parameterfor a particular unit (e.g., “What is the color-codestatus of fuel in the Battalion?”). A“needs assess-ment” question asked which of the various unitsat a particular level needed a particular resource(e.g., “What platoon in Company B needs Brad-leys?”). The participants were instructed to respondas accurately and as quickly as possible; no dis-cussion of specific strategies was provided.

A total of 18 trials were completed during ablock of trials; each block consisted of two setsof 9 trials (a factorial combination of the three


echelon levels and the three question types). Thepresentation order of the trials within a subblockwere randomized. The specific company (one offour) or platoon (1 of 12) and combat readinessparameter (one of five) that appeared in a questionwas chosen at random. Participants pointed andclicked on buttons (see Figure 3a) to indicate theirresponse. After each trial the participant was pro-vided feedback on accuracy of his or her respons-es. Thus, participants completed 324 trials (18 trialsin each of six blocks during three experimentalsessions).

RESULTS

Accuracy scores were computed as correct orincorrect. Latency was measured from the appear-ance of a question until the initial response (1/20-s

accuracy). Latency outliers were identified usingthe test described in Lovie (1986, pp. 55–56): T1 =(x(n) – x)/s, in which x(n) is a particular observation(one of n observations), x is the mean of thoseobservations, and s is the standard deviation ofthose observations. Accuracy scores associatedwith latency outliers were also not considered. Thepercentage of outlier scores was 2.24%, 1.62%,and 1.77% for the quantitative, categorical, andneeds assessments, respectively. Nonparametrictests were conducted to assess the distribution ofoutliers across experimental conditions; none wassignificant. Remaining scores were averaged acrossbattle phase, session, and repetition; a set of fivepreplanned orthogonal contrasts were performed(see Figure 5a). The results of the preplanned con-trasts involving interface are listed in Figure 5b.Additional contrasts were performed to assess the

Contrast # Verbal description:

1. Main effect – interface 1 1 1 –1 –1 –12. Main effect – echelon: battalion (B) vs. company (C) / platoon (P) 2 –1 –1 2 –1 –13. Main effect – echelon: C vs. P 0 1 –1 0 1 –14. Interaction effect – interface × echelon: B vs. C/P 2 –1 –1 –2 1 15. Interaction effect – interface × echelon: C vs. P 0 1 –1 0 –1 1

A.

Quantitative Categorical NeedsAccuracy Latency Accuracy Latency Accuracy Latency

Contrast # F p < F p < F p < F p < F p < F p <

1. Interface ns 231.39 .01 10.49 .01 79.52 .01 13.15 .01 51.36 .014. Interface × B vs. C/P 9.61 .02 179.41 .01 11.14 .01 62.98 .01 15.28 .01 ns

4a. Interface at B 8.92 .02 207.08 .01 21.24 .01 70.76 .01 ns4b. Interface at C/P ns 145.25 .01 ns 34.38 .01 16.18 .01

5. Interface × C vs. P ns ns ns 18.42 .01 ns 113.36 .015a. Interface at C 59.34 .01 112.00 .015b. Interface at P 10.10 .01 ns

B.

RA

PTO

R, B

atta

lion

RA

PTO

R, C

ompa

ny

RA

PTO

R, P

lato

on

FBC

B2,

Bat

talio

n

FBC

B2,

Com

pany

FBC

B2,

Pla

toon

Interface, Echelon:

Figure 5. Preplanned contrasts and empirical results. 5a. Preplanned, orthogonal contrasts conducted for interfaceand echelon level, including contrast weights. 5b. F-values and significance levels for the preplanned contrasts involv-ing interface and for the tests of the simple main effects of interface (following a significant interaction). A shaded cellsignifies performance advantages favoring the RAPTOR interface; ns signifies that a contrast was conducted but theresults were not significant at the .05 level. B = battalion, C = company, P = platoon.


simple main effects of interface when significantinterface by echelon interaction contrasts (Con-trasts 4 and 5) were obtained; the results are alsolisted in Figure 5b. The interface and echelon inter-action means for each of the three assessmentsare illustrated in Figure 6.

DISCUSSION

The pattern of results was clear and unequiv-ocal, indicating that the RAPTOR interface wasmore effective than the FBCB2 interface. Five ofthe six contrasts testing the main effect of interfacewere significant (Contrast1, Figure 5b); there wereseven significant interaction contrasts (Contrasts4 and 5, Figure 5b) and 10 significant contrasts forthe simple main effects of interface (Contrasts 4a,4b, 5a, and 5b). Each statistical comparison be-tween interfaces that was significant indicated thatperformance with the RAPTOR interface was bet-ter than performance with the FBCB2 interface(these contrasts are highlighted with grayscaleshading in Figure 5b; see also Figure 6). The supe-rior performance of the RAPTOR interface waspresent in all assessment categories (quantitative,categorical, and needs), dependent variables (accu-racy, latency), and echelon levels (battalion, com-pany, platoon).

These results clearly indicate that the RAPTORinterface provided better support for obtainingfriendly combat resources than did the FBCB2 in-terface. They will be interpreted from the cognitivesystems engineering perspective (Rasmussen etal., 1994) alluded to in the Introduction (see Ben-nett & Walters, 2001, for a focused discussion inthe context of display design). Specifically, over-all performance will be determined by the qualityof mapping between three sources of constraints:those constraints contributed by the domain (thedemands to be met), the agents (capabilities/limi-tations), and the interface (requirements introducedthrough design). The discussion will be organizedaround the principles of direct perception and ma-nipulation.

Direct Perception

The RAPTOR interface was specifically de-signed to support direct perception, as discussedin the Introduction section. The content mappings(domain constraints ↔ interface constraints) wereeffective: information from the various categoriesof the abstraction hierarchy were present in the

Worse0

0.8

0.9

1

45 40 35 30 25 20 15 10 5 0

Acc

urac

y (%

cor

rect

)

Latency (s)

Better

0

0.8

0.9

1

45 40 35 30 25 20 15 10 5 0

Acc

urac

y (%

cor

rect

)

Latency (s)

Worse

Better

Better

0

0.8

0.9

1

45 40 35 30 25 20 15 10 5 0

Acc

urac

y (%

cor

rect

)

Latency (s)

Worse

Figure 6. Mean levels of performance for all interfaceand echelon combinations. Accuracy is plotted on the yaxis; latency is plotted on the x axis (note that better per-formance is located in the upper right portion of thegraph). Filled and unfilled symbols identify meansobtained with the RAPTOR and the FBCB2 interface,respectively. Squares, circles, and triangles identifymeans obtained for the battalion, company, and platoonechelon levels, respectively. (A) Quantitative assessment.(B) Categorical assessment. (C) Needs assessment.

(A)

(B)

(C)


interface, as were consistent summaries of com-bat resources at all relevant echelon levels. Theformat mappings (display constraints ↔ agentconstraints) were also effective. The graphicalrepresentations were carefully designed to reflectthe inherent constraints of the domain informationbeing represented (e.g., unique representations foreach combat parameter) and to support informa-tion pickup (e.g., categorical, analogical, and al-phanumeric visual information corresponding toassessment requirements). The constraints intro-duced by the RAPTOR interface allowed skill-basedinteraction: It decreased the amount of cognitiveresources and mental effort required and allowedthe agent to use powerful visual perceptual skillsto obtain information regarding friendly combatresources.

In contrast, the FBCB2 interface did not sup-port direct perception effectively. The quality ofthe content mappings was poor. There was little re-gard for the categories of information that shouldbe present in the interface (i.e., levels of the ab-straction hierarchy). In addition, data regardingfriendly combat resources were presented in piece-meal fashion (e.g., no summarization or integra-tion across lower echelon levels). The quality offormat mappings was equally ineffective. The pri-mary form used to represent combat resources wasalphanumeric (i.e., the long-form messages) asopposed to graphical. This forces the agent to uselimited cognitive resources (i.e., working memo-ry) to derive information mentally. (For more de-tailed discussions of similar considerations, seeBennett & Flach, 1992; Bennett, Nagy, & Flach,1997; and Bennett & Walters, 2001.) As a result,acquiring information with the FBCB2 interfacerequires extensive search (i.e., navigation throughmultiple screens to locate all of the relevant data)and extensive cognitive processing (maintainingand manipulating these data in limited-capacityworking memory). In summary, although theFBCB2 interface provides an abundance of low-level alphanumeric data, there is actually very lit-tle graphical information about domain resourcesto be perceived directly.

Direct Manipulation

The RAPTOR interface provides resources thatsupport direct manipulation. Consider the quoteby Norman (1986), who described direct manipu-lation as “the qualitative feeling of control that candevelop when one perceives that manipulation is

directly operating upon the objects of concern tothe user. The actions and the results occur instan-taneously upon the same object” (p. 53). A criticalcontrol function for a commander engaged in tac-tical operations is to change his or her span of atten-tion to monitor progress and coordinate activitiesacross the organizational hierarchy (i.e., battalion,company, platoon). The commander needs to con-trol the grain of resolution at which friendly forcesare considered and to see these units in the contextof the battlefield terrain. The RAPTOR interfacesupports this need by providing icons that repre-sent these real-life objects of interest (i.e., the 17units of action) and their resources directly. Theseicons can be manipulated directly to change boththe resolution (i.e., the various units of action) andthe context (whether the icons appear on the bat-tlefield terrain) of combat resource information.

The surface appearance of the FBCB2 interfacesuggests that direct manipulation is present: thetabs, buttons, and fields are graphical objects thatcan be pointed at and clicked on. However, thissurface appearance is misleading. The graphicalrepresentations of the real-life objects of interest(i.e., the unit symbols on the contour map) cannotbe manipulated directly: Obtaining combat re-source information involves indirect manipulationof the tabs, buttons, and fields. The lack of directmanipulation resources in the interface imposesinefficient action sequences. Changing the resolu-tion of combat resource information (e.g., viewingthe resources of a lower level unit) involves re-peating the basic action sequence from scratch, asopposed to the context-conditioned shortcuts en-abled by RAPTOR (e.g., pointing and clicking acompany icon on the map). Direct manipulationcannot be used to view combat resources in thebattlefield context: The map is covered by the largedisplay windows (see Figures 3 and 4). In sum-mary, the FBCB2 interface may well qualify as a“graphical user interface”; however, the manipu-lation that it supports is far from direct.

GENERAL DISCUSSION

These results indicate that the perception-actionicons design strategy provides an effective solutionfor a critical challenge posed by one intermediatedomain, military command and control. Early the-oretical perspectives of human-computer interac-tion (Hutchins et al., 1986) emphasized the role ofdirect manipulation, to some degree at the expense


of direct perception. In contrast, theoretical per-spectives on interface design for law-drivendomains tend to emphasize the role of direct per-ception, in part because direct manipulation ofhigher-order properties is not feasible, given themany-to-many mappings and conflicting goalsthat characterize these domains. Insights from eco-logical psychology (e.g., Gibson, 1966) suggestthat direct perception and direct manipulation havea far closer and much more symbiotic relation-ship. More specifically, this work suggests that theincorporation of an intact perception-action loopshould be considered as a higher-order goal in in-terface design. The perception-action icons in theRAPTOR interface achieve this goal: The agentuses highly efficient perceptual-motor skills to pickup the affordances presented in the space-time sig-nals in the interface (direct perception of the icons)and to coordinate and synchronize execution ofcritical control functions for controlling the spanof attention (direct manipulation of the icons).

The potential for the perception-action iconsdesign strategy to generalize beyond the contextof military command and control will now be en-tertained. This strategy is a hybrid solution thatadapts and draws selectively from general strate-gies developed for domains located at the ends ofthe continuum. The perception-action icons designstrategy will be compared and contrasted withthese general strategies so that its defining char-acteristics are clear. The factors that contribute tothe success of this design strategy for the currentintermediate domain (i.e., military command andcontrol) will be described. Another intermediatedomain and the potential utility of the perception-action design strategy will then be considered.

The constraints in law-driven domains have ahigh degree of regularity that facilitates analysisand modeling; the critical design activity involvesthe mapping of domain constraints into analoggeometric representations that provide concretespatial analogies. Configural displays can be par-ticularly useful in this role, at least when designedproperly (e.g., Bennett & Flach, 1992): They willproduce higher level visual features (e.g., closure,symmetry, parallelism) and dynamic behaviorsthat accurately reflect domain constraints. Theinterface constraints that are imposed by this de-sign strategy are well mapped to powerful skill-based behaviors of the human agent (e.g., thepickup of visual information). The human agentcan “see” system states and potential solutions,

rather than deducing them. Note that the domainconstraints are inherently complex; therefore, thevisual analogies will also be rich and complex.Thus, the success of this design strategy relies upona knowledgeable and experienced human agent.

In intent-driven domains there is less regular-ity in the constraints of the domain, and thereforethe agents’ intentions and goals play a larger rolein the unfolding interaction. These users will typ-ically have far more diverse sets of knowledgeabout the domain, more diverse sets of computerskills, and less extensive experience with the de-cision support system. The use of spatial meta-phors in the interface can relate the requirementsfor interaction to more familiar objects and activ-ities, thereby leveraging preexisting concepts andknowledge. For example, the BookHouse system(Pejtersen, 1992) uses spatial metaphors exten-sively to assist agents in locating a book of fiction.The interaction requirements are related to famil-iar activities: The agent navigates through an over-arching spatial metaphor (i.e., a virtual library) toselect subsets of books (i.e., enter a different wingof the library) and to execute different search strat-egies (i.e., enter a different room). Similarly, spe-cific search terms are specified through the directmanipulation of icons with spatial metaphors that“suggest” the search term through an associativelink to preexisting concepts in semantic memory.Thus, the icons provide affordances and serve assigns that represent the various actions that can beexecuted.

The perception-action icons design strategydraws selectively from these two categories, adapt-ing the details to the context presented by militarycommand and control. First, the overlap with thedesign strategy for intent-driven domains will beconsidered. The extensive use of icons to facili-tate interaction is a key feature of both designstrategies. In both cases the icons present affor-dances and serve as signs for actions that can betaken; the icons can be manipulated directly toexecute these control inputs. Akey difference liesin the visual representations that are used in theseicons. The icons for intent-driven domains (e.g.,the BookHouse) use metaphorical representa-tions that relate interaction requirements to morefamiliar concepts and activities. In contrast, theRAPTOR icons contain a variety of representa-tions (geometrical, categorical, symbolic, alpha-numeric) that are designed to convey specific anddetailed information about the domain. Herein lies


the overlap with successful design strategies forlaw-driven domains. A unit’s combat resourcesare the tangible contributors to the overall combatpower of the unit; the value of these continuousvariables must be conveyed directly, not meta-phorically.

More fundamentally, the need for the perception-action icons design strategy appears to arise fromthe defining characteristics of the objects of inter-est in the domain of military command and con-trol. First, consider the objects of interest at thetwo end points of the continuum: The objects forintent-driven domains are essentially independentand loosely coupled (e.g., books of fiction); theobjects for law-driven domains are highly depen-dent and tightly coupled (e.g., mass balance, ener-gy balance). The objects of interest in militarycommand and control (in this case, the variousunits of action) possess both of these qualities.The units are clearly dependent and coupled (e.g.,organizational structure, coordinated missiongoals), unlike the objects of interest in intent-driven domains. At the same time, they also pos-sess a potentially high degree of independence(e.g., independent resources, independent missionroles), unlike the objects in law-driven domains.The perception-action icons design strategy issuccessful because it supports these dual needs.Information regarding combat resources can beobtained collectively or individually through thedirect perception of the icons that correspond tothe 17 units of action constituting the organiza-tional structure of the battalion. Direct manipula-tion of these icons allows the combat resources tobe considered at the proper grain of resolution andcontext for assessing progress toward collectiveor individual mission goals.

The perception-action icons design strategy islikely to be successful for other intermediate do-mains to the extent that the objects of interest inthese domains share the defining characteristicsoutlined previously. Another intermediate domainwill be examined in greater detail to explore thispossibility. Flexible manufacturing qualifies asan intermediate domain, primarily because of theincorporation of “just-in-time” production strate-gies (Rasmussen et al., 1994) that require sub-stantial discretion on the part of the operator. Arepresentative example is described by Dunkler,Mitchell, Govindaraj, and Ammons (1988). Sev-eral categories of products are manufactured; eachcategory is associated with a different set of man-

ufacturing constraints. These constraints includethe number and type of machining operations, thetemporal sequencing of these operations, and theamount of time that is required. There are a lim-ited number of configurable machining cells thatcan be used to perform the various operations.There is also an automated scheduler. However,the automated scheduler is not always capable ofproducing an acceptable solution, given the com-plex space of manufacturing possibilities (includ-ing inventory). Therefore operator intervention isoften required. In summary, there are collectivesystem goals with regard to both the category andthe number of products that need to be producedwithin a particular time frame. Meeting these goals,however, requires the consideration of individualproducts: the number, type, and sequencing of themachining operations that need to be accomplishedif the product is to be completed on schedule.

Although the surface details are different, thedefining characteristics of the objects of interest arereasonably similar to those in military commandand control, and it appears that perception-actionicons would provide a very effective interface de-sign strategy for this domain. Direct perceptioncould be achieved by designing an icon that graph-ically represents the manufacturing goals andconstraints associated with an individual product(e.g., machining operations and scheduled com-pletion times relative to production goals). The di-rect perception of these constraints would clearlyspecify instances in which the operator needs tooverride the automated scheduler to expedite theprocessing of a product that is late. In turn, the oper-ator could execute this control input through thedirect manipulation of these icons (i.e., draggingand dropping an icon on the graphical representa-tion of a machining center or processing buffer).In summary, the results of the present evaluationand an analysis of the potential for generalizationsuggest that perception-action icons constitute aninterface design strategy that will prove success-ful for other intermediate domains.

ACKNOWLEDGMENTS

We would like to thank the officers of the 4thBrigade, 85th Division (Training Support), whovolunteered to participate in this experiment. Wewould also like to the numerous U.S. Army officerswho provided their valuable input and feedbackduring the design of the RAPTOR interface. This


work was supported through participation in theAdvanced Decision Architectures CollaborativeTechnology Alliance sponsored by the U.S. ArmyResearch Laboratory under Cooperative Agree-ment DAAD19-01-2-0009. The study reported inthis article was completed in partial fulfillment ofthe requirements for the master of science degreefor Christopher Talcott.

REFERENCES

Bennett, K. B., & Flach, J. M. (1992). Graphical displays: Implicationsfor divided attention, focused attention, and problem solving.Human Factors, 34, 513–533.

Bennett, K. B., & Fritz, H. I. (2005). Objects and mappings:Incompatible principles of display design – A critique of Marinoand Mahan. Human Factors, 47, 131–137.

Bennett, K. B., Nagy, A. L., & Flach, J. M. (1997). Visual displays. InG. Salvendy (Ed.), Handbook of human factors and ergonomics(2nd ed., pp. 659–696). New York: Wiley.

Bennett, K. B., & Walters, B. (2001). Configural display design tech-niques considered at multiple levels of evaluation. Human Factors,43, 415–434.

Burns, C. M., & Hajdukiewicz, J. R. (2004). Ecological interface design.Boca Raton, FL: CRC Press.

Center for Army Lessons Learned. (1997). Initial impressions report(July, 1997): Advanced warfighting experiment (NTC rotation 97-06). Fort Leavenworth, KS: Author.

Dunkler, O., Mitchell, C. M., Govindaraj, T., & Ammons, J. C. (1988).The effectiveness of supervisory control strategies in schedulingflexible manufacturing systems. IEEE Transactions on Systems,Man, and Cybernetics, 18, 223–237.

Gibson, J. J. (1966). The senses considered as perceptual systems.Boston: Houghton Mifflin.

Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1986). Direct manipu-lation interfaces. In D. A. Norman & S. W. Draper (Eds.), User cen-tered system design (pp. 87–124). Hillsdale, NJ: Erlbaum.

Klein, G. A. (1994). A script for the commander’s intent statement. InA. H. Levis & I. S. Levis (Eds.), Science of command and control:Part III. Coping with change (pp. 75–86). Fairfax, VA: The ArmedForces Communications and Electronics Association InternationalPress.

Lovie, P. (1986). Identifying outliers. In A. D. Lovie (Ed.), New devel-opments in statistics for psychology and the social sciences (pp.44–69). London: British Psychological Society/Methuen.

Martinez, S. G., Bennett, K. B., Talcott, C. P., Stansifer, C., & Shattuck,L. (2001). Cognitive systems engineering analyses for army tacticaloperations. In Proceedings of the Human Factors and ErgonomicsSociety 45th Annual Meeting (pp. 523–526). Santa Monica, CA:Human Factors and Ergonomics Society.

Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W.Draper (Eds.), User centered system design (pp. 31–61). Hillsdale,NJ: Erlbaum.

Pejtersen, A. M. (1992). The Bookhouse: An icon based database systemfor fiction retrieval in public libraries. In B. Cronin (Ed.), The mar-

keting of library and information services 2 (pp. 572–591). London:Aslib, The Association for Information Management.

Prevou, M. (1995). The battle command support system: A command andcontrol system for force XXI [Monograph]. Fort Leavenworth, KS:U.S. Army School of Advanced Military Studies.

Rasmussen, J. (1983). Skills, rules, and knowledge; signals, signs, andsymbols, and other distinctions in human performance models.IEEE Transactions on Systems, Man, and Cybernetics, SMC-13,257–266.

Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitivesystems engineering. New York: Wiley.

Sanderson, P. M., Flach, J. M., Buttigieg, M. A., & Casey, E. J. (1989).Object displays do not always support better integrated task perfor-mance. Human Factors, 31, 183–198.

Tufte, E. R. (1990). Envisioning information. Cheshire, CT: GraphicsPress.

Vicente, K. J. (1991). Supporting knowledge-based behavior throughecological interface design (EPRL-91-02). Urbana-Champaign, IL:University of Illinois, Department of Mechanical Engineering,Engineering Psychology Research Laboratory.

Christopher P. Talcott is a professor of military scienceand leadership at the University of California, Los Ange-les. He received his M.S. in human factors and industrial/organizational psychology from Wright State Universityin 2001.

Kevin B. Bennett is a professor of psychology at WrightState University. He received his Ph.D. in applied exper-imental psychology from the Catholic University ofAmerica in 1984.

Silas G. Martinez is the executive officer of the 326thEngineer Battalion, U.S. Army, Ft. Campbell, Kentucky.He received his M.S. in human factors and industrial/organizational psychology from Wright State Universityin 2002.

Lawrence G. Shattuck is a senior lecturer of operationsresearch, Naval Postgraduate School, Monterey, Cali-fornia. He received his Ph.D. in cognitive engineeringfrom the Ohio State University in 1995.

Craig Stansifer is a human factors engineer at WrightPatterson Air Force Base, Ohio. He received an M.S. inhuman factors engineering from Wright State Univer-sity in 2006.

Date received: June 20, 2002Date accepted: December 22, 2005

perception-action icons: an interface design strategy for...

Documents