situation awareness in aircraft maintenance teams · e-mail address: [email protected] (m.r....

*Corresponding author. Tel.: #1-770-565-9859; fax: #1-770-579-1132.

E-mail address: [email protected] (M.R. Endsley).

International Journal of Industrial Ergonomics 26 (2000) 301}325

Situation awareness in aircraft maintenance teams

Mica R. Endsley!,*, Michelle M. Robertson"

!SA Technologies, 4731 East Forest Peak, Marietta, GA 30066, USA"Liberty Mutual Research Center, Hopkinton, MA 01748, USA

Accepted 15 June 1998

Abstract

Research was conducted at a major airline to investigate factors related to situation awareness in aviation maintenanceteams. Situation awareness has been found to be critical to performance and error prevention in many environments. Itsrole in the maintenance domain for the performance of both individuals and teams is discussed. Situation awarenessrequirements for aviation maintenance were determined as well as the technologies and personnel resources used toachieve situation awareness. Barriers and problems for situation awareness both across and within teams involved inaviation maintenance were revealed. Based on this analysis, recommendations for the development of a training programto improve situation awareness in aircraft maintenance at the individual and team level are presented.

Relevance to industry

The importance of situation awareness for preventing errors in maintenance is discussed as well as factors thatcontribute to problems with situation awareness across multiple teams. Speci"c recommendations for improvingsituation awareness through organization and system design and through training are made that are applicable to a widevariety of industrial settings. ( 2000 Elsevier Science B.V. All rights reserved.

Keywords: Situation awareness; Teams; Maintenance; Training; Error

1. Introduction

Insu$cient attention has been paid to humanerror in aircraft maintenance. While the number ofincidents due to mechanical failures that can betraced to maintenance problems are relativelyfew when compared to other causal factors (e.g.in#ight human error), they do exist and can besystematically addressed. Marx and Graeber

(1994), for instance, report that 12% of accidentsare due to maintenance and inspection faults, andaround one-third of all malfunctions can be at-tributed to maintenance de"ciencies. In addition toits impact on safety of #ight, the e$ciency of main-tenance activities can also be linked to #ight delays,ground damage and other factors that directlyimpact airline costs and business viability.

In examining human error that may occur withinthe maintenance arena, several key issues can beidenti"ed.

(1) The "rst involves shortcomings in the detec-tion of critical cues regarding the state of the

0169-8141/00/$ - see front matter ( 2000 Elsevier Science B.V. All rights reserved.PII: S 0 1 6 9 - 8 1 4 1 ( 9 9 ) 0 0 0 7 3 - 6

aircraft or subsystem. Several accidents havebeen traced to metal fatigue or loose and miss-ing bolts that should have been visible to main-tenance crews. Incidents exist of aircraft beingreturned to service with missing parts or in-complete repairs. Frequent errors include looseobjects left in aircraft, fuel and oil caps missingor loose, panels and other parts not secured,and pins not removed (Marx and Graeber,1994). While several factors may contribute tothis type of error, in all of these cases the stateof the system (i.e. the defect, or the loose ormissing item) was not detected prior to return-ing the aircraft to service.

(2) Often, even when important information isperceived, there may be di$culties in properlyinterpreting the meaning or signi"cance of thatinformation. For instance, Ru!ner (1990)found that in more than 60% of avionics re-pairs, the incorrect avionics system is replacedin an aircraft. While the symptoms may beobserved correctly, a signi"cant task remainsin properly diagnosing the true cause of thefailure. While not much data exists regardingthe impact of misdiagnoses of this type, there isa signi"cant increase in the probability of anincident occurring when the aircraft under-takes the next #ight with the faulty system stillaboard.

(3) Problems in properly detecting the state of thesystem and diagnosing or interpreting cuesthat are perceived are compounded by the factthat many di!erent individuals may be in-volved in working on the same aircraft. In thissituation, it is very easy for information andtasks to fall through the cracks. The presence ofmultiple individuals heightens the need fora clear understanding of responsibilities andgood communications between individuals tosupport the performance of shared tasks.

(4) In addition to the need for intra-team coord-ination, a signi"cant task for maintenancecrews is the coordination of activities and pro-vision of information across teams to those ondi!erent shifts or in di!erent geographical lo-cations. For example, an Eastern Airlines air-craft nearly crashed when oil pressure was lostto all engines almost simultaneously due to

a maintenance error in servicing the engines inwhich critical o-rings were left o! (NationalTransportation Safety Board, 1984). This errorhas been directly linked to a problem withcoordination of information across shifts andbetween maintenance departments (along withother contributing factors). In addition, con-siderable energy is often directed at coordina-tion across maintenance sites to accommodatemaintenance tasks within the #ight scheduleand part availability constraints. These factorsadd a level of complexity to the problem thatincreases the probability of tasks not beingcompleted, tasks completed improperly, im-portant information not being communicated,and problems going undetected as responsibil-ity for tasks becomes di!used.

1.1. Situation awareness

All of these di$culties point to a lack of situationawareness (SA). Situation awareness has beenfound to be important in a wide variety of systemsoperations, including piloting, air tra$c controland maintenance operations. Maintenance crewsneed support and training in ascertaining the cur-rent state of the aircraft system in addition tocurrent training programs that concentrate ontechnical skills. Formally de"ned, `situation aware-ness is the detection of the elements in the environ-ment within a volume of space and time, thecomprehension of their meaning, and the projec-tion of their status in the near futurea (Endsley,1988). In the context of aircraft maintenance, thismeans being aware of the state of the aircraft sys-tem (and the subsystem one is working on). TermedLevel 1 SA, this would include perception of factorssuch as metal fatigue, loose or missing items, pins,or screws, oil or #uid leaks, tread wear, or systemsnot functioning properly. Level 2 SA would involvethe technicians' understanding or comprehensionof the signi"cance of observed system states. Spe-ci"cally this would include their diagnosis of thecausal factors associated with observed symptoms.An aviation maintenance technician (AMT)with Level 1 SA might be aware that a particularsubsystem is not working properly. An AMT withLevel 2 SA also understands what is speci"cally

302 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Fig. 1. Team situation awareness (from Endsley, 1989a).

wrong with that subsystem. Level 3 SA, the abilityto project the state of the system in the near future,is considered the highest level of SA in dynamicsystems. An AMT with Level 3 SA would be able toproject what e!ect a particular defect might haveon the performance of the aircraft in the future.

While SA has generally been discussed in termsof the operation of a dynamic system, such as anaircraft, the concept is also applicable to the main-tenance domain. The complexity of aircraft systemsand the distributed nature of equipment and systemcomponents posses a signi"cant challenge to theAMT's ability to determine the state of the system(Level 1 SA) during diagnosis and repair activities.Putting together observed cues to form a properunderstanding of the underlying nature of malfunc-tions (Level 2 SA) is a signi"cant problem in diag-nostic activities. In the maintenance domain,AMTs may need to be able to project what willhappen to an aircraft's performance with (or with-out) certain actions being taken or with givenequipment modi"cations/repairs/adjustments oc-curring (Level 3 SA). This task may be even moredi$cult for AMTs. As they often receive little or nofeedback on the e!ects of their actions, they mayhave di$culty developing an adequate mentalmodel for making accurate predictions. The abilityto project system status forward (to determine pos-sible future occurrences) also may be highly relatedto the ability to project system status backward, todetermine what events may have led to an observedsystem state. This ability is particularly critical toe!ective diagnostic behavior.

1.2. Team situation awareness

In aircraft maintenance, as in many other do-mains, the requirement for situation awareness be-comes compounded by the presence of multipleteam members and multiple teams. Teams di!erfrom a collection of individuals in that they area distinguishable set of two or more people whointeract dynamically, interdependently, and adap-tively toward a common and valued goal/object/mission, who have each been assigned speci"c rolesor functions to perform, and who have a limited lifespan of membershipa (Salas et al., 1992). Thus, themajor factors contributing to the concept of a team

are shared goals, the interdependence of their ac-tions, and the division of labor in terms of estab-lished responsibilities for meeting those goals.Based on this framework, teams will have somedivision of labor, and thus some of their SA require-ments will be independent. However, they will alsohave shared goals and their actions will be interde-pendent, and thus they will also have shared SArequirements.

Team situation awareness can be de"ned as `thedegree to which every team member possesses thesituation awareness required for his or her respon-sibilitiesa (Endsley, 1989). In this context, the weaklink in the chain occurs when the person who needsa given piece of information (per his or her jobrequirements) does not have it. Therefore, the con-cept of team SA embodies the need for each personon a team to have the information needed forhis/her speci"c job. As shown in Fig. 1, if oneconsiders the SA requirements of di!erent teammembers, there will be some area of overlap be-tween them. That is, there will be some SA require-ments (at each of the three levels) that are pertinentonly to a given individual, but also some SA re-quirements that are the same across one or moreteam members. Successful team performance re-quires that not only does each team member havegood SA on his or her individual requirements, but

M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 303

also the same SA across shared requirements. Thishas been termed `shared SAa.

In maintenance teams, as in teams in many otherdomains, errors frequently occur because informa-tion is not successfully transmitted from one indi-vidual to the next within a team, or from one teamto the next. The failures may occur at Level 1 (in-formation not transmitted successfully to a team orteam member), Level 2 (information not compre-hended properly or in the same way across teamsor team members), or Level 3 (the implications oftransmitted information for future events notunderstood or the same across teams or team mem-bers). The ways in which these failures might occurand the factors that contribute to such failures havenot previously been examined from the perspectiveof situation awareness in the maintenance arena.

The objective of the present study was to addresssituation awareness related di$culties in aircraftmaintenance. This information would be useful forsupporting the development of cohesive mainten-ance teams and the promotion of team situationawareness. This is critical for the development of anenvironment in which people have a shared under-standing of who does what when, reducing theprobability of information or tasks going unat-tended. Individuals need to not only understandthe status of the system they are working on, butalso what other individuals or teams are (and arenot) doing as well, as both factors contribute totheir ultimate decision making and performance.

Within a team it is important to provide themembers with the skills to function e!ectively. Spe-ci"cally, training can be provided to assist teams inachieving situation awareness as this is the criticalfactor that will allow team members to carry outthe maintenance tasks they have been trained for.Several related training programs have been suc-cessful within this domain. Drury (1993) has shownsuccess in training maintenance personnel in visualinspection. Taylor et al. (1993) were successful inimproving high-level performance objectives (e.g.dependability and safety) after instituting a pro-gram to train aircraft maintenance personnel inpositive, assertive communication and teamworkcoordination skills. While both of these trainingexamples may help SA, neither directly addressesSA as an over-riding objective. In particular, other

factors need to be considered to optimize team SA.For example, skills are needed for identifying criti-cal information and ensuring that it is passedacross teams (and team members) and then inter-preted based on a common framework across teammembers. The present research therefore sought toidentify SA requirements for various aircraft main-tenance teams, analyze how SA needs are currentlybeing met in a typical maintenance environment,and establish concepts and requirements for train-ing to improve SA in maintenance teams.

Why examine the maintenance domain from theperspective of SA? First, based on a naturalisticmodel of decision making, we "nd that peoples'classi"cation of the situation largely drives theirdecision making and performance (Endsley, 1995b;Klein, 1989). Studies of human error in the cockpithave found that the vast majority of errors are duenot to poor decision making, but to problems withSA (Endsley, 1995a; Hartel et al., 1991). The presentstudy speci"cally sought to determine whether SAwas relevant in the maintenance domain and theway in which it serves decision making and perfor-mance in this, a very di!erent, environment.

Secondly, examining SA may provide a uniqueperspective on the challenges and problems withinthe maintenance arena. The reasons for many er-rors and shortcomings in aircraft maintenance maynot be apparent at "rst glance. By looking at SA,and the many challenges to SA in this domain, abetter understanding of the causes of human errormay be achieved. It is important to note that whilemany problems may occur due to challenges ininformation acquisition (e.g. noise, poor lighting,out-dated technology), the way in which informa-tion is used and understood by individuals withinthe maintenance organization, across its variousteams, shifts and locales, may also underlie manyproblems in the system.

2. Methodology

Two major research activities were conductedtowards the accomplishment of the researchgoal: (1) A determination of the requirements andresources for SA in aircraft maintenance teams, and(2) an assessment of training needs for improving


Fig. 2. Team SA context analysis.

team SA. The objective of the requirements analysiswas to determine exactly what SA means within themaintenance environment: What is it that needs tobe perceived by each person in his or her job, whatis it that needs to be comprehended and whatprojections need to be made? Based upon an under-standing of the actual requirements for SA in vari-ous parts of the organization, recommendationscan be made for improving the technologies ororganizational factors available for supporting SAand for creating a training program to enhance SAin aviation maintenance teams. While SA is criticalfor decision making and performance in manyareas, developing appropriate methods for improv-ing SA in any given domain is contingent on suchan analysis.

Since it was not possible to review practices atall airlines or all locations, this research was con-ducted at an aircraft maintenance facility for onemajor US airline which served as a representativemaintenance environment. Speci"cally, B-checkmaintenance operations at a major airport wereanalyzed. The B-check is conducted on a periodicbasis for all aircraft and involves thorough checksof avionics systems and power units, variousinterior and exterior components, and lubricationand checks of engines, wings and gears.

The subject airline is a large US air carrier. It wasbelieved that the selected airport B-check operationwas fairly representative of the type of activitiesperformed in aircraft maintenance, although obvi-ously variations may exist due to di!erences be-tween airlines or management practices at variousmaintenance stations. This airline, like many in theindustry, had undergone a number of reorganiza-tions in the previous "ve years which involvedconsolidating maintenance bases and out-sourcingsome of its maintenance functions. While a signi"-cant impact to personnel within the organizationand to the way of doing business, it was not felt thatthese factors would signi"cantly a!ect the majorgoals of the study } determining the applicability ofSA and team SA to aircraft maintenance, and de-signing intervention strategies for improving it.

The research was conducted by "rst identifyingSA requirements and the resources used to supportthose requirements in the selected representativemaintenance environment and operations. Concepts

for training team SA were developed based on theresults of the analysis. A team SA context analysismethodology was developed for conducting theanalysis. This method consists of two parts: An SArequirements analysis and an SA resource analysis,as shown in Fig. 2.

2.1. SA requirements analysis

The "rst step was to determine the speci"c situ-ation awareness requirements of individuals in theaircraft maintenance arena. This was addressedthrough a goal-directed task analysis which as-sessed (1) the goals and sub-goals associated withmaintenance crews, (2) the decision requirementsassociated with these goals, and (3) the situationawareness requirements necessary for addressingthe decisions at all three levels } detection, compre-hension, and projection. This type of analysis hasbeen successfully conducted for pilots in severalclasses of aircraft (Endsley, 1989,1993), air tra$ccontrol (Endsley and Jones, 1995; Endsley andRodgers, 1994) and airway facilities maintenance(Endsley, 1994).

Analyses were conducted through expert elici-tation with experienced maintenance personnel,observation of aircraft maintenance activities, andreview of all available maintenance documentation.The analysis concentrated on B-Check mainten-ance activities. Interviews were conducted withthree maintenance supervisors, four lead techni-cians and four AMTs. In addition, personnel in


planning and stores, maintenance control, mainten-ance operations control and aircraft-on-ground in thecompany's technical operations were interviewed.

2.2. SA resource analysis

The second part of the team SA context analysisconcentrated on identifying the SA resources usedin the maintenance environment to achieve the SArequirements identi"ed in the goal-directed taskanalysis. Two major categories of resources wereconsidered: other personnel as a source of informa-tion and technologies used as sources of informa-tion.

To provide an assessment of the personnel SAresources, an analysis of communications betweenorganizations and individuals was conducted usinga contextual inquiry approach. The contextualinquiry approach (O'Neill and Robertson, 1996;Robertson and O'Neill, 1994) focuses on under-standing and describing the communication pat-terns within and between teams as related to theirperformance and job goals. The contextual in-quiries were conducted simultaneously with theinterviews for determining the SA requirements.The contextual inquires involved semi-structuredinterviews in which each individual was asked todescribe his/her major functions and goals and theorganizations or individuals that served as re-sources in meeting those goals. A mapping wasdetermined showing the communications and in-teractions among and between team members.Each individual was asked to make an estimate ofthe overall frequency of communication with eachidenti"ed unit and the importance of the commun-ication for achieving their goals. Finally, they wereasked to identify barriers to e!ective communica-tion and performance in the work setting.

In addition, the technologies for obtaining eachrequirement within the current system were doc-umented. Based on this analysis, an assessment wasmade of the degree to which the current system sup-ports SA in maintenance teams and the skills andabilities that are required for achieving good SAwithin this environment. This assessment was usedto identify system design recommendations andtraining concepts for improving SA in maintenanceteams.

3. Results

A hierarchy of goals in the maintenance environ-ment was developed for several categories of jobfunction within the maintenance team (supervisors,leads and AMTs), and for several organizationsor teams that work closely with the maintenanceteam to achieve its goals: Material Services (stores),Planners, and Maintenance Control, includingMaintenance Operations Control and Aircraft-on-Ground (an organization that deals with acceler-ated maintenance for aircraft that do not meetairworthiness requirements). These goals were usedto develop a list of SA requirements for each group.Next, the personnel resources used for meetingthese SA requirements were determined. Thetechnology resources used for meeting the SArequirements were also surveyed and barriers andproblems for achieving job goals were identi"edduring the interviews. As these analyses were quiteextensive, the results for only the AMT are shownhere as an example. The analysis of SA issues acrossmaintenance teams will also be discussed.

3.1. Goals and functions

Speci"c tasks and functions included within theB-check operations were identi"ed to include:(1) avionics in cockpit: radar, radios, #aps, gauges,(2) interior: lavatories, emergency equipment, sea-ts, overhead bins, lap belts, emergency lighting(3) exterior: tires, brakes, fuselage, leading edgesand #aps, cargo bays, (4) right and left engines andwings, (5) right/left gear lubrication, nose gear lu-brication and tail lubrication, (6) auxiliary powerunit, and (7) addressing placards (notices placed inmaintenance logs for non-functioning items, allow-ing the plane to be #own for a speci"ed number ofhours using redundant systems while repairs arecompleted).

An analysis of the AMTs' goals in performingtheir tasks revealed that they appear to be orientedtowards the dual objectives of ensuring aircraftsafety and delivering aircraft for service on time.A breakout of AMT goals is shown in Table 1. Ingeneral, the top level goals of supervisors and leadsare very similar to those of AMTs shown in Table 1.They assume many of the same subgoals as the


Table 1AMT goals

1.0 Aircraft safety1.1 Deliver aircraft in airworthy, safe condition

1.1.1 Find potential problems1.1.2 Solve problems1.1.3 Make repairs

1.1.3.1 Determine part availability1.1.3.2 Placard problem

1.1.4 Service aircraft1.1.5 Provide quality workmanship

1.2 Keep area clean2.0 Deliver aircraft on time

2.1 Prioritize tasks

AMTs in identifying and solving maintenanceproblems as needed to support AMTs when theyrun into di$culties. Supervisors and leads, how-ever, also have additional subgoals associated withmanaging the maintenance teams (assigning andprioritizing tasks, assessing aircraft status and pro-viding coordination).

Reviewing the goals of each of the other organ-izations involved in aviation maintenance revealedsigni"cant interdependencies between teams. Forinstance, AMTs are dependent on material servicesto have parts, tools, and materials ready whenneeded. Material services is in turn dependent onthe planners to provide relevant task informationand on the aircraft-on-ground organization to de-liver needed parts as soon as possible. These inter-dependencies, while not surprising, highlight theneed for good information transfer across teams.By examining in detail the SA requirements of eachteam, it should be possible to gain an understand-ing of the types of information required by eachteam and the ways in which the transfer of thisinformation can be improved.

3.2. SA requirements

A breakout of SA requirements for each goal andsubgoal identi"ed for each maintenance team wasderived from the interviews and observations. Thegoals, subgoals, major decisions and SA require-ments for AMTs are shown in Table 2. The analysisidenti"ed a number of SA requirements at all threelevels (perception, comprehension and projection)

that are important for meeting goals in this do-main. In reviewing Table 2, a few issues should benoted.

(1) At any given time more than one goal or sub-goal may be operating, although these goalswill not always have the same priority. The SArequirements in Table 2 do not assume anyprioritization among them, or that each sub-goal within a goal will always be present.

(2) These are goals or objectives not tasks. Theanalysis is as technology free as possible. Howthe information is acquired is not addressed inthe SA requirements analysis. (This matter isaddressed in the SA resource analysis).

(3) The analysis sought to de"ne what AMTswould ideally like to know to meet each goal. Itis recognized that they often must operate onthe basis of incomplete information and thatsome desired information may not be easilyavailable with today's system.

(4) Static knowledge, such as procedures or rulesfor performing tasks, is also outside the boundsof this analysis. The analysis primarily identi"-es dynamic situational information (informa-tion that changes from situation to situation)that e!ects how AMTs perform their tasks.

A similar analysis was conducted for each of theother maintenance teams (e.g. stores, inspectors,etc.) This analysis shows the factors that are impor-tant for SA in maintenance crews, not only in termsof the basic information that is needed, but alsohow that information is used, interpreted and


Table 2AMT SA requirements

1.0 Aircraft safety1.1 Deliver aircraft in airworthy, safe condition

1.1.1 Find potential problemsf Item within or beyond serviceable limits?f Item near limits needing preventive maintenance?

f reported problemsf pilot reportsf placards

f new problemsf worn tires/brakesf miswiringf dents/damagef loose itemsf fuel/oil leaksf items out of ordinaryf functioning of convenience items

1.1.2 Solve problemsf Fix problem or defer?

f potential impact of problem on #ight safetyf time required to solve problem

f time required to get partf length of time item can be deferred without repairf location(s) aircraft is going to

f facility maintenance capabilitiesf today's loadf problem deferability category (placardable, groundable)

f minimum equipment list (MEL) statusf How to solve problem?

f impact of potential approaches on timef impact of potential approaches on #ight safetyf impact of potential approaches on other tasks/jobs

f possible methodsf possible sources of problemf maintenance/failure history of itemf part availability (see 1.1.3.1)

f proposed repair authorizedf Engineering Change Request Authorization

number

1.1.3 Make repairs1.1.3.1 Determine part availability

f Correct part supplied?f manufacturer's part numberf aircraft type, model, tail numberf maintenance and equipment list numberf e!ectivity number

f How long to get part here?f in-stock status

f manfacturer's part numberf aircraft type, model, tail numberf maintenance and equipment list numberf e!ectivity number


Table 2 (Contd.)

f part & tooling availabilityf wheref when it will be heref delivered or pick-upf arrival #ight numberf arrival gate number

1.1.3.2 Placard problemf Can problem be placarded?

f type of problemf Minimum Equipment List

(MEL) statusf Deferred information placard (DIP)f Open item list (OIL)f redundant systems availablef control numberf log page numberf #ight numberf employee number

1.1.4 Service aircraftf Service activities needed?

f tasks to be donef fuel statusf lavatory statusf Are we meeting schedule?

f time aircraft due at gatef delays to aircraft

f estimated time of arrival at gatef aircraft repair status

f Where do we need to go?f gate assignmentsf permission to taxif permission to do high power run-upf taxi/runway clearances

f Current status of job?f status of other tasks impacting own taskf other tasks own task will impactf who can helpf who needs help

f tasks startedf tasks completedf tasks/activities being done nextf who is doing each taskf activity currently being performed by others

f major problems encountered1.1.5 Provide quality workmanship

f Activities performed correctly?f tasks performed correctly

f steps to be donef steps completedf location of designated components on systemf system typef paperwork completedf parts installed correctlyf inspection approved


Table 2 (Contd.)

1.2 Keep area cleanf Area free of foreign objects?

f loose objects (screws, parts, 2), tools, trash2.0 Deliver aircraft on time

2.1 Prioritize tasksf Best order for tasks?

f task time requirementsf interdependence/sequencing requirements of tasksf part availability (see 1.1.3.1)f problem deferability category (placardable, groundable)

f minimum equipment list (MEL) statusf availability of kits, tools, equipment, vehiclesf availability of personnel, skill level

Table 3AMT personnel SA resources

SA resources: personnel Meanimportance

Meanfrequency (%)

Supervisor 3.2 2.25Maintenance control 2.5 3.75}Maintenance operations Control}Aircraft on ground (AOG)Quality assurance 2.5 (1.00Aircraft inspectors 2.0 3.25Planning 2.0 (1.00Lead technician 1.5 26.75Stores 1.5 8.25Other AMTs 1.0 54.50Airport operations 1.0 (1.00Company operations 1.0 (1.00

analyzed to make decisions and carry out requiredtasks. Any attempts to develop technologies orother changes to support the maintenance techni-cian must directly impact on the accuracy or ease ofachieving each of these requirements.

3.3. SA resources: personnel

The personnel resources and technology re-sources used within the organization to meet theSA requirements identi"ed in Table 2 were ascer-tained through the contextual inquiry methodo-logy. Results of the contextual inquiries for theAMT are presented in Fig. 3. The "gure depicts thepersonnel SA resources, in terms of the individuals

or units within maintenance technical operations,that are needed to achieve the AMTs' SA require-ments and job goals. Lines and arcs show commun-ication patterns between organizations.

Estimates were made by each individual who wasinterviewed concerning the overall frequency ofcommunication with each of the other maintenanceorganizations and other team personnel. Each indi-vidual assigned a number to each organizationwhich re#ected the overall frequency of commun-ication of each interchange (out of a total of 100%).For each of these SA resources, the importance ofthe communication also was rated on a four-pointscale, where 1 represents a very important resourcefor achieving the team's performance goals and4 represents a relatively low importance resource.Mean estimates of communication frequency andimportance were determined for each of the main-tenance organizations.

Table 3 displays the personnel SA resources,mean importance and mean frequencies for thecommunication interfaces from the perspective ofthe AMT. Several personnel and work units wereindicated as very important SA resources necessaryto accomplish the AMT's job. These were the otherAMTs, airport operations (tower) and companyoperations (ramp personnel), closely followed byleads and stores (material services). The highestreported frequency of communication was with theother AMTs (54.50%), followed by lead technicians(26.75%).

Overall, a great deal of interdependency wasfound between the organizations and personnel


Fig. 3. AMT personnel SA resources.

included in this study. Each job type involved inter-acting with between 3 and 14 di!erent organiza-tions to attain (or supply) the information neededfor good SA and job performance. In general, twoor three of these interactions were viewed as veryimportant and constituted the majority of eachorganization or team's interactions, however, formany of the teams there were also many organiza-tions that were interacted with only occasionally.Speci"c issues regarding these interactions that canhave a signi"cant impact on SA were uncoveredduring the analysis and presented in the discussionsection.

3.4. SA resources: technologies

In addition to ascertaining the personnel re-sources used for attaining situation awareness,the technologies used within the maintenanceorganization were also examined. The primarytechnologies used for passing information includeda commercial information system for logging main-

tenance activities, several non-integrated databasesused across several organizations, and technicaldocumentation which could be found in varioushard copy manuals and micro-"che. (Issues andrecommendations involving these technologies areaddressed later.)

3.5. Barriers and problems

During the interview process, factors that createdbarriers to e!ective communication and perfor-mance were elicited. Barriers are issues that slowdown or hinder performance. These are problemsthat maintenance teams must routinely overcomein order to meet their goals. They encompass or-ganizational, technical and personnel issues. Ingeneral, most people felt that the system workedquite well, however, almost all could name a fewareas where improvement was possible. These arelisted in Table 4.

The most frequently mentioned barrier wasa lack of proper tooling for completing jobs, a


Table 4Barriers to performance

Barrier Frequency

Lack of tooling; out-sourcing of parts 13Parts availability; determining status of parts 10No backlog of critical parts 10Non-integrated databases; redundancy of tasks;hard to "nd needed information

10

Tracking the parts; getting the parts to the aircraft 8Computerized database 8Lack of support and feedback from management 8Other organizations don't understand what we do,problems we face

8

Lack of teamwork; information being passedamong & between team members

7

Personality con#icts 7Instability of organization 7Downsizing of organization 6Shiftwork; fatigue 6Computer system in stores 5Workcards; changing of procedures with aircraft 5Need for better information and communicationto solve problems locally

5

Streamline engineering authorizations 4Poor housekeeping and maintenance of tools 4Computer system for customer service 3Need more training on using computer system 3Need for more proactive procedures 3Need to develop consistent procedures forobtaining/borrowing parts

3

Need more explicit requirements for contractsuppliers

3

Low experience levels of some personnel dueto lay-o!s and job changes

2

problem which is exacerbated by the fact that muchof the repair work has been out-sourced to outsidecontractors, making rapid access to correct toolingand parts di$cult. In addition to a certain lack oftrust of these external organizations (probablystemming from lack of information), there was anexpressed frustration with not being able to interactwith personnel where the repair work had beenout-sourced so that questions pertaining to speci"citems could be addressed. A decreased reliability ofparts and quality control problems with partscoming from out-sourced vendors was also men-tioned. Maintenance personnel expressed the needto be able to track parts by vendor names and to

track the quality and reliability of these parts whichthe current system does not support.

The second most commonly listed problem wasan unavailability of parts and di$culty in deter-mining when the proper parts would be available tothe AMT. Often the parts supplied would not becorrect for the speci"c model and type of aircraft.This is a particular problem as the company hasaircraft which were purchased from many di!erentairlines, each with subtle di!erences between them.Parts supplied by stores often are not the correctones due to these slight di!erences. This serves asa frequent sources of frustration, necessitating sche-dule delays or issuing a placard for repair at a laterdate. Related to this problem is the lack of a back-log of critical parts. Critical parts are frequently notavailable when needed, leading to having an air-craft down for an extended period or necessitatingexpensive and time consuming rush procurementsthrough the aircraft-on-ground organization.

Tracking parts for a speci"c aircraft was fre-quently mentioned as a signi"cant di$culty. Deter-mining where parts are in the system (speci"cally inrelation to items being obtained from outside thesystem, or in transit from somewhere) and gettingthe parts to the aircraft were described as commonproblems. In general, maintenance personnel ex-perienced signi"cant uncertainty regarding when,where and how parts would be delivered and spentextra time trying to get this information and toascertain its reliability.

Signi"cant problems in switching between thevarious information databases (such as the storesdatabase and the customer service database amongothers) were noted. Maintenance personnel cur-rently need to retrieve information from multiplesources, however, the ability to readily access andgain needed information from multiple systems atthe same time is quite limited. For example, book-ing, monitoring bills, baggage handling, and track-ing items and parts are all activities that need to beconducted by aircraft-on-ground. These activitiesrequire accessing and integrating informationacross several databases on an almost continualbasis in order to keep up-to-date with the currentsituation. This situation also leads to redundanttasks between paperwork, manuals and the com-puter systems. Maintenance personnel expressed


a need for an integrated computerized databasesystem, allowing for more e$cient monitoring ofactivities and parts thus facilitating getting the partto the aircraft in a more expeditious fashion. Otherneeds expressed included a ready list of `hota partsand items, a means of tracking minimum equip-ment list (MEL) items better and a database onparts reliability. The commercial computer in-formation database in particular was considereda signi"cant barrier or problem. Personnel ex-pressed considerable frustration with the system asit made data entry very di$cult and the user inter-face was very clumsy.

Organizational issues were also mentioned. Inparticular, the feeling was expressed that manage-ment was not providing visible and active support,particularly in regard to feedback on how person-nel were doing, improvements that could be made,and guidance on which direction personnel shouldgo in and why. Maintenance personnel expresseda desire for better feedback or rewards when theymake progress in the right direction.

Maintenance personnel also expressed a certaindegree of frustration regarding other organizations.Many felt that other groups did not really under-stand what they did. For example, the AMTs didnot like having maintenance operations control(MOC) tell them what to do, when `they aren't outhere working in the cold and the darka. On theother hand, MOC personnel felt they were misun-derstood as they all had worked in the AMT jobbefore. They also felt they had the best informationto be able to ascertain the impact of a given prob-lem on changes in the system (e.g. scheduling). Theydid not feel the AMTs had this `big picturea. Theend result of these types of di!erences is misunder-standings between organizations, and ine$cienciesin problem solving as neither group has the fullpicture and the same information possessed by theother group.

Several interpersonal issues were mentioned.While most personnel were considered to be `teamplayersa, others were considered to be de"cient bynot pitching in to help complete tasks. Problemswith information not being transferred betweenteam members both during a shift and betweenshifts were cited. Related to this, personnel con#ictswere listed as a problem. The instability of the

organization was also a signi"cant concern. Justprior to the time period of this study, there weremany reorganizations, changes in management,layo!s, and reassignments/relocation of many ofthe personnel.

Other problems mentioned included fatigue andproblems associated with shift work (particularlyamong graveyard shift workers), concerns over or-ganizational down sizing, lack of up-dating of thestores computer system to re#ect the nuances ofparticular aircraft, a need for more training on thecomputer system, a lag in updating work cards tore#ect changes in work procedures, and poorhousekeeping and maintenance of tools.

People expressed the desire to be able to solveproblems locally if only they had the informationthey needed. For example, a particular problemmay be placarded and passed on to another station,when it could be "xed locally if information onscheduling and parts availability was shared better.People at the local level wanted to be more in-volved in the decision-making process in order tohelp meet the organizational goal of having theaircraft back in service as soon as possible. Relatedto this issue, personnel also expressed a desire formore proactive problem solving instead of waitinguntil a crisis situation develops. They felt theyneeded to get information sooner and to obtainearlier involvement of the respective parties in theproblem solving process.

A need was also expressed for streamlining theprocess used for obtaining engineering authoriz-ations and for developing consistent proceduresthat everyone could follow for borrowing and ob-taining parts. Due to a lack of consistent proced-ures, a lot of time and e!ort may go into oneparticular strategy for obtaining a part and thenwhen that method falls through, they have to trya new strategy having lost valuable time. Proced-ures that incorporate alternate parallel tracks andaction plans when parts are needed are desired.Di$culty with contract suppliers was expressed.The feeling was that contract suppliers need to begiven clearer expectations regarding what they areto deliver and quality requirements. Clearer pro-cedures and processes need to be conveyed to them,particularly in light of signi"cant culture and timezone di!erences.


Finally, the low experience levels of some person-nel was described as a problem. Due to a number oflayo!s, people with high seniority, but perhaps lowexperience level in a particular job type are morecommon. This has a signi"cant impact on schedul-ing the AMTs on particular tasks and teams. Withthe perceived pressure to save money and do morewith less, this issue was felt to have an impact onperformance as more highly trained individualsmight not be able to work with fellow team mem-bers as much as might be needed.

4. Discussion

Based on the analysis of the SA requirements andresources for each organization and the barriersand problems expressed, several observations canbe made pertinent to team SA in the aircraft main-tenance domain. The largest problem for team SAexists when gaps are present between organizationsor individuals. These gaps may be the result ofmismatched goals, lack of needed information onthe part of one or both parties, lack of understand-ing of the exact information that another groupneeds, or di!erent interpretations of informationthat is passed from another group.

AMTs face several challenges in meeting their SArequirements that can be linked to team SA. First,AMTs spend a great deal of their time and re-sources in ascertaining whether they have the cor-rect parts or when and how they will get the correctparts. A considerable gap exists between the AMTsand the stores organization which often may supplythe incorrect part (due to di$culties with e!ectivitynumber di!erences between di!erent aircraft mod-els, for instance) or may not have the correct partdue to stocking limitations. These situations in-crease both the probability of error (incorrectlyinstalling the wrong but very similar part) and maylead to considerable ine$ciencies, waste and de-lays. When parts are not available, the AMTs fre-quently must involve their leads and supervisors,maintenance control and aircraft-on-ground. Thisnecessitates the involvement of several organiza-tions and personnel, all of whom need to bebrought up to speed on pertinent situational in-formation to make good decisions. This process is

time consuming and may be prone to miscommuni-cation errors, leading to SA problems.

The process of placarding also poses a signi"cantproblem. AMTs may spend a considerable amountof time disassembling a system and trouble-shoot-ing to arrive at a diagnosis, only to "nd they cannot"x the problem due to an unavailability of parts orschedule constraints. This is a process which isfairly ine$cient and which they "nd very frustrat-ing due to lack of closure in addressing the veryproblems they are trained to "x. Completing re-pairs is a factor from which they derive their majorjob satisfaction. AMTs get very discouraged whenthey are not allowed to "x things that clearly need"xing. It is also a waste of time and human re-sources to have to reassemble a subsystem andplacard it so that it can be unassembled again and"xed later on at another maintenance station. Al-though sometimes placarding is unavoidable, it isgenerally best if problems can be "xed immediately.The system does not appear to be currently opti-mized to avoid placarding, however. A review ofthe goals of supporting organizations, such asmaintenance control and its sub-organizations, re-veals that they place far more emphasis on rem-edying existing placards than avoiding new ones.This goal mis-match may be at the heart of con-siderable misunderstandings between groups.

While AMTs report a need to ascertain jobstatus and schedule progress, they currently getonly limited information concerning these issues.While they supply information regarding progresson their own tasks up the line on an ongoing basis,leads and supervisors frequently provide little in-formation back down the line over the course ofa shift. Leads reportedly did not feel that AMTsreally needed information on how other team mem-bers were doing in terms of progress on their re-spective tasks. Without this knowledge, however,the AMTs have no way to engage in compensatoryactivities (e.g. pitching in to help each other), andmay not be aware of ongoing activities of otherteam members that may have an e!ect on their owntasks (or vice versa). In some cases, tasks must bedone in a certain order. In other cases, certain taskscan a!ect the activities of other AMTs in a way thatcreates a safety hazard unless both parties takeprecautionary measures. Thus, a lack of up-to-date


knowledge on within team progress contributes toSA gaps within the maintenance team.

Although regulations are very speci"c regardingthe criteria specifying when an item must be re-paired or replaced, discretion is available in allow-ing AMTs to repair or replace items that might benearing the acceptable limit. It may be both saferand more time and cost e!ective to promote thistype of action in some circumstances (e.g. if a givensubsystem is already disassembled for other workand the part is available). Discretion is also avail-able on items that are placardable: they can be "xedimmediately or placarded and sent to another sta-tion for later repair. Better sharing of information isneeded in regard to these issues so that AMTs,leads and supervisors can make decisions that arein line with de"ned organizational priorities andrealities. For example, they may need the informa-tion that a given aircraft will only be going to otherstations that are not well equipped to "x the par-ticular type of problem or that are overloaded. Thiswould indicate that they should "x the probleminstead of placarding it, even it means a slightschedule delay. Personnel need to understand whatcurrent organizational priorities are and why.A shared understanding of the cost and bene"ttradeo!s in "xing things on the spot versus delay-ing repairs would allow them to form a betterunderstanding of situations they encounter andmake better decisions.

In reviewing the SA requirements and resourcesof the maintenance leads and supervisors, it isapparent that they serve largely in the role ofcoordinators and can become information`middle-mena. In addition to administrative duties,they become involved when problems arise andassistance is needed, providing support themselvesor interacting with other organizations (e.g. main-tenance control or aircraft-on-ground) to getneeded support. This role is very critical in theprocess of achieving good SA at the team level.When they become involved, supervisors and leadsneed to get a considerable amount of situationalinformation from the AMTs or from others whomay be in geographically distributed locations.This process can be highly prone to informationfalling through the cracks or can result in in-dividuals not forming a full understanding of a

situation. If supervisors do not have a completeunderstanding of all pertinent information, for in-stance, they may not pass information on that willallow maintenance control to make the best deci-sion. In addition, leads and supervisors are fre-quently responsible for passing information back tothe AMT. If they only pass information regardingwhat the AMT should do (the decision) but notregarding why the decision was made, this may leadto both a lack of understanding by the AMT andmay deny the AMT the opportunity to volunteerinformation he or she may have that would bepertinent to the decision made. Leads and supervi-sors form a critical link in the SA chain between thevarious organizations and need to have a full un-derstanding of what information other peoplereally need and of how to get all the informationthey need themselves.

Stores (material services) appears to work prim-arily based on planned demands in order to obtainneeded parts in advance. Some stores personnel,however, reportedly do not understand the uniquedi!erences between aircraft models and tail num-bers that allow them to procure parts with theproper e!ectivity number. (This problem appearsto be at least partially due to problems in documen-tation and the databases provided to them.) Thissituation leads to considerable problems withAMTs who complain of not having the correctparts. The lack of availability of needed parts hasbeen identi"ed as one of the most critical factors indetermining whether an aircraft will be repaired ornot. There also exist problems in keeping up withthe status of the inventory when there are numer-ous people who have access to parts and may notkeep databases up-to-date. The greatest SA needfor this group is in determining methods to insurethat they have correct information on needed partsand to provide them with a better ability to projectparts requirements (Level 3 SA). While they dowork with projections from planning and with typi-cal part usage requirements, their ability to projectrequirements for parts could probably be enhancedthrough better system feedback and advancedplanning.

Maintenance control (MC) and its sub-groupsappear to function largely in a trouble shooting,reactive mode. They become involved when help is


needed and primarily focus on expediting problemsolutions by bringing resources (parts, expertise,routing) to bear on identi"ed problems. They faceseveral challenges in this role. Maintenance controlhas a great deal of general system knowledge at itsdisposal, both in terms of technical skill and docu-mentation. AMTs in the "eld have the best situ-ation knowledge, however, as they are on-site withthe aircraft and have the most contact with perti-nent aircraft data. The challenge is to combinethese two sources of information most e!ectively toarrive at proper diagnoses and solutions. This gapbetween those with situation information and thosewith the best technical knowledge may be reducedwith improved understanding between the twogroups or from technologies that assist in the shar-ing of information between the two groups.

Although the stated goal of maintenance opera-tions control is to minimize placards, it shouldbe noted that they primarily focus on makingsure existing placards do not exceed prescribedtime limits. The process seems to be to "rst approveplacards (if allowable) and then to work to removethem. Neither maintenance operations control normaintenance control appear to focus on proactivetasks to avoid placards in the "rst place. This stateof a!airs also appears to form a gap between theseorganizations and the AMTs in the "eld. It may bethat some organizational streamlining betweenmaintenance operations control and maintenancecontrol may also be of bene"t, reducing the need tohave distributed decision making in meeting theirshared goal.

In reviewing the SA resources used by eachgroup, several general comments can also be made.It appears that the AMTs interact mostly withother team members on site. Moving up in theorganization, leads and supervisors are far morelikely to interact with other groups (such as plan-ning, stores, and maintenance control) and withmaintenance units at other stations, as do the sup-port organizations. These groups have an increasedneed to understand the other groups they interactwith. For example, understanding the di!erencesbetween maintenance sites (manpower availabilityand skill levels, load levels, parts and equipmentavailability) may be very important in allowingpersonnel to develop a good understanding of the

impact of decisions or to understand why otherorganizations are making certain statements. Theseissues are very important to e!ective decision mak-ing, particularly when organizations are highly geo-graphically distributed and such di!erences maynot be evident. Each maintenance site also needs tobe able to share relevant information (e.g. problemsdetected with certain aircraft or parts) across sitesin order to allow the whole organization to achievethe highest knowledge level possible. Across theorganization, members of each group need to de-velop an understanding of what information isneeded by other groups and how to clearly pass onneeded information about their own situations.

5. System and organizational recommendations

The issues unearthed by this analysis are myriad.Clearly, many di!erent strategies can be proposedthat would be helpful for improving information#ow and SA within the organization. Table 5 showsseveral changes and improvements to both thetechnologies and organizational system that can beimplemented to improve the distribution of in-formation. While these factors are speci"c to thisairline, many of these proposed solutions may alsobe applicable to the maintenance organizations ofother airlines.

Improving situation awareness within a widelydistributed organization such as aviation mainten-ance, however, will also require that personnel areable to properly utilize this information to formgood higher level assessments of the situation (com-prehension and projection) and are better ableto communicate information within and acrossmaintenance teams. For this reason, a program fortraining to improve team SA within aviationmaintenance was also developed.

6. Team SA training recommendations

Several training concepts were identi"ed for im-proving Team SA within the maintenance settingbased on this analysis and discussion.


Table 5Technological and organizational interventions

Information system improvements

f Support for information exchange with out-sourced maintenance organizations and vendorsf Support to insure proper parts are available and supplied for jobsf Support for tracking parts on orderf Integration of multiple databases used within the maintenance organizationf Support for tracking MEL items, parts reliability information, and &&hot'' itemsf Redesign of user interface/usability of computer support system

Organizational and personnel improvements

f Better feedback and guidance from managementf Better understanding between maintenance organizationsf Better teamworkf Improved organizational stabilityf Improved use of consistent procedures for engineering authorizations and obtaining partsf More training and use of experienced personnelf Organizational streamliningf Increased involvement by personnel at all levels in problem solving

6.1. Shared mental models

From the analysis it was determined that di!er-ent teams (organizations) do not have a good men-tal model of what other teams know, do not knowor need to know. Good situation awareness at theteam level depends on having a clear understandingof what information means when it is conveyedacross team members. Thus, teams need to sharenot only data, but also higher levels of SA, includ-ing the signi"cance of data for team goals andprojection information. This process is greatly en-hanced by the creation of a shared mental modelwhich provides a common frame of reference forteam member actions and allows team members topredict each other's behaviors. A shared mentalmodel may provide more e$cient communicationsby providing a common means of interpreting andpredicting actions based on limited information,and therefore may form a crucial foundation fore!ective teamwork. When shared mental modelsare not present, one team may not fully understandthe implications of information transmitted fromanother team and misunderstandings, errors andine$ciencies are likely to occur. By providing eachteam with better information on the goals of otherteams, how they perform their tasks, and what

factors they take into account in their decisionprocesses, a better shared model can be developed.This should greatly enhance not only interpersonalinteractions among teams, but also the quality ofthe decision processes.

6.2. Verbalization of decisions

There also exists a need for teams to do a betterjob of passing information to other teams regardingwhy they decide to (or not to) take a particularcourse of action (e.g. deferments, schedules, etc2).Unless the rationale and reasons are passed along,considerable misunderstandings may occur. In ad-dition, this will deny the possibility of getting betterinformation from the other team, who may haveaccess to other pertinent information that wouldmake for a more optimal solution. Conveying whya particular decision was made provides a muchgreater level of SA (particularly at the comprehen-sion level). It allows other teams to either under-stand and accept the decision or to o!er othersolutions that may be better in achieving organiza-tional goals. More information also needs to getconveyed on what diagnostic activities have beenperformed when passing aircraft to another station,and a need exists for better communications


between stations and teams in general. Trainingthat focuses on teaching people to verbalize therationale behind decisions and provide greater de-tail regarding diagnostic activities should help im-prove team SA considerably.

6.3. Better shift meetings and teamwork

Team leads need to receive explicit training onhow to (1) run a shift meeting to convey commongoals for the team, (2) provide a common groupunderstanding of who is doing what, (3) set-up anunderstanding of the inter-relationship betweentasks and personnel activities and (4) provide ex-pectations regarding teamwork. Shift meetings pro-vide an excellent opportunity to provide this sharedunderstanding among the members of a team. Thisinformation is crucial for allowing team membersto have a good mental model regarding what every-one is doing and how tasks inter-relate so that theycan have good SA regarding the impact of theiractions and tasks on other personnel and on theoverall goal. Team leads also need to receive speci-"c training on the importance of passing informa-tion on job status within teams over the course ofthe shift. Without this type of feedback, people caneasily lose sight of how they are progressing inrelation to the other tasks being performed. Thisfeedback is important for individual performanceand SA, and also for fostering a team spirit incarrying out activities.

6.4. Feedback

Currently, personnel receive little feedback onhow well a particular solution worked. A trickydiagnosis and repair may have been totally success-ful, or may have failed again a few days later atanother station. At present, it is very di$cult totrack the performance of a particular action or part(partially due to the cumbersome nature of thecomputer system). Such feedback is crucial to thedevelopment of better mental models of the tech-nical systems AMTs work on. Without such feed-back, it is very di$cult to improve one's diagnosticskills. While system enhancements are recom-mended to help with this problem, it is also re-commended that people be trained to provide such

feedback. Not only do managers and leads need totake an active role in providing this feedback, butAMTs (and others) can also be trained to providemore feedback (either over the phone or throughthe computer system) on what worked and whatdid not.

6.5. SA training

Common problems can be linked to SA failuresin a number of systems, including (1) forgettinginformation or steps, frequently in association withtask interruptions, (2) not passing information be-tween shifts or team members, (3) missing criticalinformation due to other task related distractions,and (4) misinterpreting information due to expec-tations. Training can be used to provide heightenedawareness of these problems and ways of combat-ing them. For instance, task interruptions area common problem leading to SA errors. Fre-quently, such interruptions lead to skipping stepsor missing activities. Personnel can be trained totake particular measures following a task interrup-tion (double check previous work performed,double check area for loose tools, etc.). This type oftraining may be useful for helping maintenancepersonnel to insure that they are not missing criti-cal information in the performance of their tasks.

6.6. Team SA training program

The team SA training course (Endsley andRobertson, 1997; Robertson and Endsley, 1997)was developed to address these "ve SA traininggoals and objectives. In addition, the course alsoprovides a review of maintenance resource manage-ment (MRM) principles which are considered to beprior knowledge requirements for the trainees. Theteam SA training course was designed to be pre-sented as an 8 h classroom delivery to personnelfrom across all maintenance operations depart-ments. The course is best taught to a classcomposed of a mixed cross section from di!erentmaintenance operations organizations (e.g. stores,AMTs, inspectors, maintenance operations control,etc.). This is because the course focuses on helpingto reduce the gaps and miscommunications thatcan occur between these di!erent groups and it was


anticipated that much of the course's bene"t wouldcome from the interaction that occurs whentrainees share di!erent viewpoints and informationin going through the exercises.

An extensive set of Powerpointt slides coveringthe "ve team SA training principles, group exer-cises, maintenance examples and case studies areincluded as part of the course to encourage activelearning. The instructional strategy used for thecourse features adult inquiry and discovery learn-ing. This allows a high level of interaction andparticipation amongst the trainees creating an ex-periential learning process. The team SA trainingcourse strongly encourages participation in prob-lem solving, discussion groups and responding toopen ended questions, thus promoting the acquisi-tion and processing of information.

7. Evaluation of team SA training course

In order to assess the e!ectiveness of the course,an initial evaluation of it was conducted at thesame subject airline.

7.1. Evaluation method

The team SA training course was delivered bythe airline at four of its large maintenance bases.Most of the maintenance organization personnel inthis airline had already received MRM trainingwhich is considered to be a precursor to the teamSA training course. The course was delivered overa two-day period by this airline. (It was expandedfrom the original 8 h course design by this airline toallow for more group exercises, interaction andcase studies.)

Seventy-two people from nine di!erent mainten-ance locations attended the training sessions atwhich the present evaluation took place. Participa-tion in the course was voluntary and participationin the course evaluation was also voluntary andcon"dential. Participants were present from a fullcross-section of shifts. The majority of the partici-pants were male (86%), however, 14% of the par-ticipants were female. The participants came froma wide range of technical operations departmentsand job titles. The most frequent job title was that

of line mechanic (AMT), followed by leads andsupervisors. A good cross sections of other organ-izations within the maintenance organizationwere also represented, including inspection,planning, and documentation support personnel.Attendees were very experienced at their jobs(mean"10.41 yr) and within the organization(mean"12.16 yr).

The evaluation process consisted of three levels:Value and usefulness of the training, pre/post train-ing measures, and changes in behavior on the job.A questionnaire was administered immediately fol-lowing the course to get participant's subjectiveopinions on the value and usefulness of the course.In addition, the amount of learning on attitudesand behaviors related to SA was also measured.An evaluation form was provided immediately priorto the training to assess knowledge and behaviorsof the trainees related to SA. It was administeredagain immediately following the course to measurechanges in attitudes and self-reported intentions tochange behavior as a result of the training. Theform was administered again 1 month later to as-sess changes in behavior on the job.

7.2. Results

7.2.1. Value and usefulnessThe post-training course evaluation was used to

measure the level of usefulness and perceived valueof the course. Course participants scored each sub-section of the course on a "ve-point scale whichranged from 1-waste of time to 5-extremely useful.On average, they rated each of the major topics as`very usefula (mean scores between 3.5 and 4.7). Inaddition to rating topics in the course, participantsalso answered several questions related to thecourse as a whole, shown in Fig. 4. The mean ratingfor the course overall was 4.3, corresponding tobetter than `very usefula. A whopping 89% of theparticipants viewed the course as either `very use-fula or &&extremely useful'', representing a high levelof enthusiasm for the course. There were no lowratings of the course as a whole. Over 94% of theparticipants felt the course was either `very usefulaor `extremely usefula for increasing aviation safetyand teamwork e!ectiveness (mean rating of 4.4).Over 89% felt the course would be either very or


Fig. 4. Overall course evaluation.

extremely useful to others (mean rating of 4.3).When asked to what degree the course would a!ecttheir behavior on the job, 83% felt they wouldmake a `moderate changea or a `large changea, asshown in Fig. 5.

7.2.2. Changes in behavior and attitudesThe mean change in the post-test compared to

the pre-test on each behavior described in thepre/post self-reported SA behavioral measure formwas also assessed. A factor analysis on the ques-tionnaire revealed a moderate degree of homogen-eity. That is, responses on the items were somewhatinterrelated, however, no large groupings of relatedfactors were revealed to explain a large portion ofthe variance. (Only one factor accounted for morethan 10% of the variance, with most accounting forless than 5%). The questionnaire was thereforetreated as a set of independent items. Changeson each item were compared for each subjectusing a paired comparison analysis (pre-test topost-test).

The Wilcoxon non-parametric statistical analy-sis revealed that attitudes and self-reported behav-iors changed signi"cantly on seven of the 33 items(p(0.05). Participants reported after the trainingthey would be more likely to keep others up-to-date with their status as they go along in doingtheir jobs (an increase of 15%). They also wereslightly more likely to report that they would try tokeep up with which activities others were workingon over the course of the shift (an increase of 10%).Both of these items relate to improved situationawareness across the team.

Participants reported they would be more likelyto try to understand others' viewpoints when en-gaged in a disagreement with other departments(an increase of 15%). This relates to an e!ort todevelop better shared mental models regardingother departments. In addition, participants re-ported changes in several behaviors related to im-proved communications and teamwork. They weremore likely to report improved written commun-ication when sending an aircraft with a MEL to


Fig. 5. Perceived a!ect of course on behavior.

another station (an increase of 21%). Participantswere more likely to report that they wouldmake sure to pass on information about an aircraftand work status to the next station (an increaseof 13%).

They were also more likely to report making sureall problems and activities are discussed duringshift meetings (an increase of 11%), and encourag-ing others to speak up during shift meetings tovoice concerns or problems (an increase of 12%).

These di!erences between the pre-test and post-test measures on SA related behaviors and attitudesindicates that in addition to participants respond-ing positively to the course, they reported actualchanges in behaviors they would make on the jobas a result of the course, thus improving SA on thejob both between and within maintenance teams.

7.2.3. Changes on the jobIn order to assess whether participants actually

made the intended changes in their job behaviorsfollowing the course, the same form was againadministered one month following the course. Atthe time of this analysis, the participants of onlyone course had been on the job for a full monthafter the training session. Of these participants (17),six responses were available for this analysis (rep-resenting a return rate of 35% which is typical ofmail-in questionnaires). A paired comparison of

responses on each item between the post-test ques-tionnaire and the one-month questionnaire wasmade using the Wilcoxon test. This analysis re-vealed no changes on any of the test items at a 0.05level of signi"cance. Therefore, it would appear thatthe behaviors participants reported they would en-gage in following the training were carried out inpractice, at least for this small sample.

7.3. Discussion of training evaluation

Overall the SA team training course was highlysuccessful. The course content associated with all ofthe major training objectives was rated very highlywith the vast majority of participants rating eacharea as `very usefula or `extremely usefula. Thecourse was viewed as between `very usefula and`extremely usefula overall, for increasing aviationsafety and in terms of usefulness to others. Thecourse training methods and media, including thecase studies, videos and group exercises, wereviewed as particularly successful and supportive inacquiring the learning objectives. In fact, the onlysuggestion many participants had for improvementwas to use even more of these materials. Clearly, aninstructional strategy that emphasized experientiallearning and participation was e!ective for achiev-ing the training objectives and facilitating the learn-ing process.


Fig. 6. Comparison of team SA training course usefulness to CRM/MRM courses.

The course was administered to a fairly experi-enced aviation maintenance group who representeda wide range of departments and skill areas withinmaintenance department of the airline. The factthat the course included such a mix of participantsalso was viewed as a key ingredient in its success.The mixed group allowed people from di!erentareas to better understand each other's viewpoints,contributing to the development of shared mentalmodels and open communications for future deci-sion making.

The majority of participants felt that the coursewould result in making changes in their behaviorson the job. The results of the follow-up question-naire, administered one month after the trainingcourse, supported these intentions. The self-re-ported behavior follow-up questionnaire showedthat participants were making the changes they hadintended to make following the training.

These results are very similar to those achievedin previous evaluations of MRM training programswhich have been shown to be highly successful inimproving safety and performance in aviation

maintenance. Fig. 6 illustrates the enthusiast sup-port for crew resource management (CRM) andMRM courses by #ight operations and main-tenance participants, respectively, as measuredimmediately following training (Taylor andRobertson, 1995). This is compared to the responsemeasured in this study to the team SA trainingcourse. Nearly two-thirds of the #ight operationsgroups reported that the CRM training was `veryusefula or `extremely usefula (Helmreich et al.,1987). Even though this response is very strong, theresponse of maintenance personnel to the MRMtraining was even stronger. Ninety percent of themaintenance personnel sampled at two di!erentairlines felt the course was `very usefula or `ex-tremely usefula (Robertson et al., 1995; Taggart,1990). The team SA training course, evaluated inthis study, drew a response that was comparable tothat found for the highly successful MRM Trainingprogram that was conducted at the same airline(Taylor and Robertson, 1995). Based on this result,it can be concluded that the team SA trainingcourse is viewed as highly useful at a rate that is


favorably compared to previous courses in theMRM/CRM area.

These "ndings are based on the responses of aninitial group of course participants. To further vali-date these "ndings, this evaluation should be con-tinued with succeeding groups of trainees in thecourse. In addition, more follow-up research isneeded to validate the results of the on-the-jobbehavior changes. At the time of this analysis, veryfew course participants had been back on the jobfor one full month. Therefore, the sample size forthis analysis was very small; Probably, too small formuch con"dence in the results. By following upwith the remaining participants at the one monthpoint (and again at longer durations), more reliableresults can be obtained regarding the degree towhich the training e!ected job behaviors relatedto SA.

Finally, it would be highly desirable to ascertainthe degree to which the training impacts criticalmaintenance performance measures at the airline.The bottom-line objective is to reduce maintenanceerrors, improve aviation safety and improve perfor-mance. Since the course had been administered toso few participants (scattered over 9 cities), makingany meaningful assessment of the e!ect of the train-ing on performance outcomes was not feasible inthis study. In the future, however, the e!ect of thetraining implementation on several key safety andperformance measures should be assessed.

8. Conclusion

Overall, the applicability of the concept and im-portance of situation awareness in maintenanceteams has been supported in this analysis. Teams ofAMTs are supported by many other personnel andorganizational units to achieve their goals, each ofwhich has a major impact on the attainment of thethese goals. In this context it is necessary to exam-ine how information #ows between and amongteam members in order to identify system and per-sonnel factors that will impact on the degree towhich team members are able to maintain an accu-rate picture of an aircraft's status. This informationappears to be crucial to their ability to performtasks (as each task is interdependent on other tasks

being performed by other team members), theirability to make correct assessments (e.g. whethera detected problem should be "xed now or plac-arded for later), and their ability to correctly projectinto the future to make good decisions (e.g. timerequired to perform task, availability of parts, etc.).

Results from the SA requirements analysis con-ducted here provide a "rm foundation for identify-ing the knowledge, skills, and abilities thatmaintenance personnel need to attain a high levelof SA. The SA requirements identi"ed provide in-formation on the speci"c knowledge that mainten-ance personnel need to achieve a high level of SAfor completing their tasks. Providing personnelwith knowledge is not enough, however. Mainten-ance personnel must also have the skills and abil-ities required to e!ectively communicate thatknowledge, and need the ability to recognize whichinformation needs to be exchanged among andbetween team members. Several gaps betweenteams were discussed.

In addition to specifying the role of SA in anaircraft maintenance environment, an assessmentwas made of systems and technologies used tosupport SA in this organization, and potentialareas for improvement identi"ed. The training con-cepts proposed here also show promise as a meansof enhancing the skills and abilities needed forachieving a high level of SA in a team environment.These concepts were used to create a deliverabletraining program that was prototyped by a majorairline. By enhancing the SA skills of the variousmaintenance teams, speci"cally focusing on areaswhere SA breakdowns are likely to occur, improve-ments in the e$ciency and safety of the aircraftmaintenance operations can be achieved.

The prototype implementation of the course wasused to obtain an indication of the value of trainingof this nature. The idea of using training to improveSA is very new and no courses for training team SAhave previously been developed. The evaluationpresented here represents an initial evaluation ofthe team SA training course in its prototype imple-mentation phase. It was the "rst time the coursehad been o!ered to a group of maintenance person-nel. The fact that it was viewed so positively asuseful to maintenance operations is highly indica-tive of its success. It is strongly recommended that


the airline continue to implement the course andthat additional airlines consider adopting thecourse to improve SA within maintenance organ-izations. Overall, the value of the team SA trainingprogram has been supported by this analysis andfurther implementation and evaluation of thecourse are recommended.

Acknowledgements

This work was performed under a sub-contractto Galaxy Scienti"c Corporation, Dr. William John-son, Project Managers, for the Federal AviationAdministration O$ce of Aviation Medicine, Dr.William Sheppard and Ms. Jean Watson, ProgramManger. We express our sincere appreciation toMs. Karin Porter, Mr. John Stelly and Mr. TomKirwan at Continental Airlines for their supportof this research and their innovative e!orts toimprove technical operations in aviation.

References

Drury, C., 1993. In: Corporation, G.S. (Ed.), Training for visualinspection of aircraft structures. Human Factors in AviationMaintenance } Phase Three, Progress Report (DOT/FAA/AM-93/15), O$ce of Aviation Medicine. Federal AviationAdministration, Washington, DC, pp. 133}154 (Chaptereight).

Endsley, M.R., 1988. Design and evaluation for situation aware-ness enhancement. In: Proceedings of the Human FactorsSociety 32nd Annual Meeting, Santa Monica, CA, pp. 97}101.

Endsley, M.R., 1989. Final report: situation awareness in anadvanced strategic mission. NOR DOC 89-32, NorthropCorporation, Hawthorne, CA.

Endsley, M.R., 1993. A survey of situation awareness require-ments in air-to-air combat "ghters. International Journal ofAviation Psychology 3 (2), 157}168.

Endsley, M.R., 1994. Situation awareness in FAA Airway Facili-ties Maintenance Control Centers (MCC). Final Report,Texas Tech University, Lubbock, TX.

Endsley, M.R., 1995a. A taxonomy of situation awareness errors.In: Fuller, R., Johnston, N., McDonald, N. (Eds.), HumanFactors in Aviation Operations, Avebury Aviation. AshgatePublishing Ltd, Aldershot, England, pp. 287}292.

Endsley, M.R., 1995b. Toward a theory of situation awareness.Human Factors 37 (1), 32}64.

Endsley, M.R., Jones, D.G., 1995. Situation awareness require-ments analysis for TRACON air tra$c control. TTU-IE-95-01, Texas Tech University, Lubbock, TX.

Endsley, M.R., Robertson, M.M., 1997. Team situation aware-ness training program. In: Human Factors Issues in AircraftMaintenance and Inspection '97 CD-ROM. Galaxy Scienti"cCorporation, Atlanta, GA.

Endsley, M.R., Rodgers, M.D., 1994. Situation awareness in-formation requirements for en route air tra$c control. In:Proceedings of the Human Factors and Ergonomics Society38th Annual Meeting, Santa Monica, CA, pp. 71}75.

Hartel, C.E., Smith, K., Prince, C., 1991, De"ning aircrew coord-ination: searching mishaps for meaning. Paper presented atthe Sixth International Symposium on Aviation Psychology,April.

Helmreich, R.L., Foushee, H.C., Benson, R., Russini, W., 1987.Cockpit management attitudes: exploring the attitude be-havior linkage. Aviation, Space and Environmental Medi-cine 57, 1198}1200.

Klein, G.A., 1989. Recognition-primed decisions. In: Rouse,W.B. (Ed.), Advances in Man}machine Systems Research.JAI Press, Inc, Greenwich, CN, pp. 47}92.

Marx, D.A., Graeber, R.C., 1994. Human error in aircraft main-tenance. In: Johnston, N., McDonald, N., Fuller, R. (Eds.),Aviation Psychology in Practice. Avebury, Aldershot, UK,pp. 87}104.

National Transportation Safety Board, 1984. Aircraft AccidentsReport, Eastern Air Lines, Inc., L-1011, Miami, Florida,May 5, 1983, Author, Washington, DC.

O'Neill, M.J., Robertson, M.M., 1996. A participatory designprocess for developing ergonomic o$ce design criteria. In:Brown, O., Hendrick, H.W. (Eds.), Human Factors in Or-ganizational Design and Management } V. North-Holland,Amsterdam, pp. 209}215.

Robertson, M.M., Endsley, M.R., 1997. Development of a situ-ation awareness training program for aviation maintenance.In: Proceedings of the Human Factors and ErgonomicsSociety 41st Annual Meeting, Santa Monica, CA, pp.1163}1167.

Robertson, M.M., O'Neill, M.J., 1994. Development of contex-tual inquiries for examining knowledge workers in technolo-gical o$ce environments. Herman Miller, Inc., Zeeland, MI.

Robertson, M.M., Taylor, J.C., Stelly, J.W., Wagner, R., 1995.A systematic training evaluation model applied to measurethe e!ectiveness of an aviation maintenance team trainingprogram. In: Proceedings of the Eighth International Sym-posium on Aviation Psychology, Columbus, OH, pp.631}636.

Ru!ner, J.W., 1990. A survey of human factors methodologiesand models for improving the maintainability of emergingarmy aviation systems. US Army Research Institute for theBehavioral and Social Sciences, Alexandria, VA.

Salas, E., Dickinson, T.L., Converse, S., Tannenbaum, S.I., 1992.Toward an understanding of team performance and training.In: Swezey, R.W., Salas, E. (Eds.), Teams: their training andperformance. Ablex, Norwood, NJ, pp. 3}29.

Taggart, W., 1990. Introducing CRM into maintenance training.In: Proceedings of the Third International Symposium onHuman Factors in Aircraft Maintenance and Inspection,Washington, DC, pp.


Taylor, J.C., Robertson, M.M., 1995. The e!ects of crew resourcemanagement (CRM training in airline maintenance: resultsfollowing three year's experience. NASA Ames ResearchCenter O$ce of Life and Microgravity Sciences andApplications, Mo!ett Field, CA.

Taylor, J.C., Robertson, M.M., Peck, R., Stelly, J.W., 1993.Validating the impact of maintenance CRM training. In:Proceedings of the Seventh International Symposium onAviation Psychology, Columbus, OH, pp. 538}542.


situation awareness in aircraft maintenance teams · e-mail address: [email protected] (m.r....

Documents