Theses - Daytona Beach Dissertations and Theses

Spring 1995

Standardized Checklists versus Variable Checklist: An Evaluation Standardized Checklists versus Variable Checklist: An Evaluation

in a Light Twin Simulator in a Light Twin Simulator

Veronica Terese Cote Embry-Riddle Aeronautical University - Daytona Beach

Follow this and additional works at: https://commons.erau.edu/db-theses

Part of the Aerospace Engineering Commons, and the Aviation Commons

Scholarly Commons Citation Scholarly Commons Citation Cote, Veronica Terese, "Standardized Checklists versus Variable Checklist: An Evaluation in a Light Twin Simulator" (1995). Theses - Daytona Beach. 36. https://commons.erau.edu/db-theses/36

This thesis is brought to you for free and open access by Embry-Riddle Aeronautical University – Daytona Beach at ERAU Scholarly Commons. It has been accepted for inclusion in the Theses - Daytona Beach collection by an authorized administrator of ERAU Scholarly Commons. For more information, please contact commons@erau.edu.

STANDARDIZED CHECKLISTS VERSUS VARIABLE CHECKLISTS:

AN EVALUATION IN A LIGHT TWIN SIMULATOR

by

Veronica Terese Cote

A thesis submitted to the

Faculty of the

Department of Aeronautical Science

in partial fulfillment of the requirements

for the degree of

Master of Aeronautical Science

Embry-Riddle Aeronautical University

Daytona Beach, Florida

Spring 1995

UMI Number: EP32114

INFORMATION TO USERS

The quality of this reproduction is dependent upon the quality of the copy

submitted. Broken or indistinct print, colored or poor quality illustrations

and photographs, print bleed-through, substandard margins, and improper

alignment can adversely affect reproduction.

In the unlikely event that the author did not send a complete manuscript

and there are missing pages, these will be noted. Also, if unauthorized

copyright material had to be removed, a note will indicate the deletion.

®

UMI UMI Microform EP32114

Copyright 2011 by ProQuest LLC All rights reserved. This microform edition is protected against

unauthorized copying under Title 17, United States Code.

ProQuest LLC 789 East Eisenhower Parkway

P.O. Box 1346 Ann Arbor, Ml 48106-1346

Copyright by Veronica Terese Cote 1995

All Rights Reserved

STANDARDIZED CHECKLISTS VERSUS VARIABLE CHECKLISTS:

AN EVALUATION IN A LIGHT TWIN SIMULATOR

by

Veronica Terese Cote

A thesis submitted to the Faculty of theDepartment of Aeronautical Science

in partial fulfillment of the requirements for the degree of

Master of Aeronautical Science

SUPERVISORY COMMITTEE:

in

ACKNOWLEDGMENTS

I wish to express special thanks to the Thesis Chairperson, Dr. John Wise, whose

constant encouragement, helpful counsel and Opractical suggestions were crucial to the

successful outcome of this thesis. Sincere appreciation is also due to my committee

members Dr. Jefferson Koonce for his expertise in statistical methodology and Dr. Daniel

Garland for his strong support. I also wish to thank my colleagues at the Embry-Riddle

University Center for Aviation and Aerospace Research, especially Marylin Shedden, for

their assistance in various ways in this research.

My graduate work would not have been possible without the love and support of

my husband, Philip, who shared all my achievements and disappointments throughout this

experience. I am eternally grateful for his steadfast belief in me. The birth of my

daughter, Alexandra Rose, added joy to my life and determination to finish this endeavor.

I would also like to express my sincerest appreciation to my sister, Maryte

Bizinkauskas, whose ability to "see through the clouds" was invaluable during the editing

process of this thesis, and my mother, Veronica Bizinkauskas, who continuously gave me

a lift when I needed it most.

I would finally like to thank my father, the late Dr. Peter A. Bizinkauskas, for his

inspiration to never stop pursuing education. This work is lovingly dedicated to him.

ABSTRACT

Author: Veronica Terese Cote

Title: Standardized Checklists versus Variable Checklist:

An Evaluation in a Light Twin Simulator

Institution: Embry-Riddle Aeronautical University

Degree: Master of Aeronautical Science

Year: 1995

The purpose of this study was to determine if a checklist that varied by sequence

would enable the pilot to detect potential errors more easily than those who used an

unchanging checklist. A flight hour based stratified sample of pilots were randomly

divided into two groups and flew a series of eight flights in a light twin aircraft simulator.

The control group used the same checklist for each trial; the experimental group used a

checklist that covered the same items but varied in sequence for each trial. Faults were

introduced in the last two trials. The number of faults discovered or missed and the time

required for each subject to perform the pre-departure checklist was recorded. The results

indicated that there was significant difference in time between groups and over for trials

(1-6). There was no significant difference between groups after a fault was introduced in

trials seven and eight. There was no difference in error rate.

TABLE OF CONTENTS

ACKNOWLEDGMENTS iv

ABSTRACT v

LIST OF TABLES AND FIGURES viii

1. INTRODUCTION 1

Statement of the Problem 2

Review of Related Literature 3

Purpose of checklists 3

Design and use of the checklist 5

Checklist content 7

Checklist initiation and interruption 9

Psychological factors 11

Human information processing 12

Memory 13

Expectancy theory 15

Automatic and controlled processing 17

Checklist complacency 19

Hurry-up syndrome 22

Summary 23

vi

Statement of the Hypothesis 24

2. METHOD 25

Subjects 25

Equipment 26

Procedure 28

3. RESULTS 31

Errors 32

Effect of Checklist Design on Time 32

4. DISCUSSION 35

Observations 35

Analysis 40

5. CONCLUSION 42

Recommendations For Further Research 42

REFERENCES 44

APPENDIX

A Standard ATC-810 checklist 47

Variable checklists 48

B Informed Consent Form 57

C Trip Sheets 59

D List of Random Faults 62

E Raw Data 64

F Flight Hours, Simulator Hours of Subjects 73 Checklist Completion Times and Faults Missed 75

vn

LIST OF TABLES

Table 1. Pilot Hours Stratification 26

Table 2. List Of Faults During The Trials 30

Table 3. Total Error Rate For Both Groups 33

Table 4. Effect Of Checklist Design On Time 33

Table 5. A Three-Factor Mixed Design-Repeated Measures On Two-Factors Analysis Of

Variance 34

Table 6. List Of Randomly Set Faults 38

LIST OF FIGURES

Figure 1. Graph of the times of the control and experimental groups 32

vm

Introduction

The purpose of checklists has been to alleviate the burden of pilots which comes

from trying to remember all the steps necessary to configure the aircraft for various flight

regimes. The use of standardized checklists began about the time of the US Airmail

Service and evolved to a complex written list of actions to be performed, a system which

has not changed in concept from those early days despite the modern computerized

checklists.

The checklist is a critical tool for ensuring safe and consistent flight operations.

Consistent, accurate use of the checklist is a safeguard to ensure that the aircraft is

properly configured, operations are completed sequentially and efficiently, the aircraft is

prepared for flight, and the pilot is cognizant of potential problems before leaving the

ground.

The importance of checklists have long been recognized to have such a significant

impact on safety that checklists are required to be accessible and used by the pilot under

Federal Aviation Regulations (FAR) FAR Part 135, Air Taxi Operators and Flight Rules

and FAR Part 121, Airline Operations and Flight Rules. The FARs require the checklist to

include: starting engines check, takeoff check, cruise configuration check, approach check,

after landing check, and the shutdown check. The FARs also require a checklist for

1

2

emergency operations of: fuel, hydraulic, electrical, and mechanical systems, emergency

operations of instruments and controls, engine inoperative procedures, and any other

emergency procedures necessary for a safe flight (FAR/AIM, 1994). However, the

Federal Aviation Regulations do not require the use of checklists for FAR Part 91,

General Operating and Flight Rules.

Given the importance of correct checklist use, Degani and Weiner (1993) have

done considerable research into the use of checklists for airline crews. However, their

research did not consider the implication of checklist use for general aviation pilots.

Statement of the Problem

While some research into checklist procedures has been done in the context of

commercial airline operations, there has been little research with direct relationship to

general aviation. Clear procedures are established for major and commuter airlines, but

only recommended in a cursory manner in the typical general aviation aircraft. General

aviation, i.e., light aircraft flying, is generally done by a single pilot, usually with no formal

cockpit resource management (CRM) training or the benefits of strict procedural

indoctrination. Solo flight operations require diligence in checklist procedure because of

lack of redundancy of another crew member or advanced automation. Therefore, the

design of a checklist for this type of operation takes on added importance.

3

Review of Related Literature

Purpose of Checklists

The checklist has been devised over the years as a method of ensuring that critical

items necessary for the safe operation of complex systems are performed in a sequential

and logical order. It has been used in aviation extensively and has been the foundation of

pilot standardization and cockpit safety for years (Degani and Weiner, 1993). It provides

a method of verification of system components and proper aircraft configuration. The

checklist is an important backup for pilots to aid in helping them remain focused on the

task at hand and eliminate guesswork when attention may be divided during periods of

high workload or periods of stress and fatigue (FAA, 1995).

The normal checklist is intended to achieve the following objectives:

1. Provide a standard foundation for verifying aircraft configuration that will attempt to defeat any reduction in the flight crew's psychological and physical condition. 2. Provide a sequential framework to meet internal and external cockpit operational requirements. 3. Allow mutual supervision (cross-checking) among crew members. 4. Dictate the duties of each crew member in order to facilitate optimum crew coordination as well as logical distribution of cockpit workload. 5. Enhance a team concept for configuring the plane by keeping all crew members "in the loop". 6. Serve as a quality control tool by flight management and government regulators the flight crews (Degani and Weiner, 1993, p. 347).

The proper use of checklists is intended to prevent unsafe practices, carelessness

and the development of individual procedures (USAir, 1991). Another objective of the

4

checklist is to foster a positive attitude toward the use of this procedure (Degani and

Weiner, 1993). A checklist can only be effective if the pilot is fully aware of the

importance of the conscientious use of the procedure.

The National Transportation Safety Board (NTSB) has recognized the importance

of checklist use and its critical role in flight safety when it called for an intensive checklist

review following a 1969 crash of a Pan American Boeing 707 (NTSB recommendation A-

69-012).

The Federal Aviation Administration Office of Integrated Safety Analysis

conducted a review of aircraft accidents from the period of 1983-1993. The investigation

revealed that approximately 279 aircraft accidents occurred during operations conducted

under Federal Aviation Regulations Part 91, 135 and 121. These accidents resulted in

approximately 215 fatalities and 260 injuries which were a direct result of a situation

where a checklist was not used or the checklist failed to include critical steps required for

safe operation (FAA, 1995).

In a period of about two years, four highly visible accidents took place in which

incomplete or inaccurate use of the checklist was found to be contributing factor. These

deadly errors include: the failure of a flight crew of a Jetstream 31 to properly apply

maximum takeoff power which resulted in a crash immediately after takeoff (NTSB,

1988b), a rudder mis-trim that contributed to the crash of a US Air Boeing 737-400 from

La Guardia Airport (Aviation Week and Space Technology, 1990, April 2) and incorrectly

set flaps or slats which led to the crash of both a Northwest Airlines MD-80 flight 255 at

Detroit Metro Airport (NTSB, 1988a) and a Delta Airlines Boeing 727 flight 1141 from

Dallas-Fort Worth (NTSB, 1989). These accidents prompted further study of the human

factors implication in checklist design and usage (Degani and Weiner, 1990).

Design and use of the checklist

Checklist devices used in air transport or military applications include scroll,

mechanical, electromechanical, video display and paper checklists. The scroll checklist

consists of a narrow strip of paper that scrolls vertically between two reels and is most

commonly used in military aircraft. The mechanical checklist consists of a plastic slide

that is used to cover items completed. The electromechanical checklist is presented as an

internally lighted display in the cockpit in which the accomplished item is extinguished by a

toggle switch located beside each item (Degani and Weiner, 1990). The new generation

of aircraft that has cathode-ray tube (CRT) displays have the capability of presenting the

checklist on the CRT and then highlighting the item that must be accomplished (Palmer

and Degani, 1992).

The most frequently used checklist in all aviation operations consists of a printed

card with two columns. The left-hand column contains the items that needs to be checked

visually and/or configured (e.g., "FLAPS;" "GEAR HANDLE"). The right-hand column

contains the standard response to the challenge (e.g., "SET"; "DOWN-THREE

GREEN"). This design is used because it is easily updated, reproduced and simple to use.

The disadvantages are: a lack of a marker-type system to distinguish whether or not an

item has been completed, lack of a system to alert pilots of a missed item, the need of the

pilot to hold or position the checklist in a manner that allows it to be easily followed but

6

does not interfere with the required pilot action and the difficulty of reading the checklist

in low ambient light situations (Degani and Weiner, 1990).

There are two basic philosophies of checklist design and use. The first is the

"challenge and response" method, which is most commonly used in airline operations. In

this method, the pilot first configures the aircraft from memory then the written checklist

is used to verify that the aircraft is properly configured. During verification, the pilot calls

out the item, then both pilots visually confirm the status of the item. The copilot then

responds a standard answer. This method provides redundancy in that both pilots become

actively involved in confirming the configuration of the aircraft and the checklist reader

confirms the appropriate response. This method, when carefully performed by the flight

crew, provides the highest degree of accuracy.

The second method of checklist use is referred to as a "do-list" which is most

commonly used by general aviation pilots. In this method, the pilot reads the item and

then performs the action. If the sequence is interrupted or incomplete, items may missed

and mistakes may go unnoticed (Degani and Weiner, 1993). Because the pilot does not

first configure the aircraft from memory, this step by step procedure does not provided the

redundancy of the challenge and response method. A single pilot operation is at an added

disadvantage because the experience and aid of another pilot precludes the increased level

of safety redundancy provides.

7

Checklist content

There is considerable debate as to which items should be on a checklist. Some

argue that most of the configuration items should be included because the procedure

should verify that all required items. Others argue that because a checklist is redundant to

the pilot's initial configuration, only the critical items should be included. Because of the

differing philosophies, the exact make-up of the checklist will vary between aircraft and

between companies flying the same aircraft (Degani and Weiner, 1990).

The use of on-board computers in sophisticated commercial and corporate aircraft

has allowed monitoring of aircraft systems. Reliance on automation for aircraft

configuration warnings is not without its unique problems. Although both the Northwest

flight 255 and Delta flight 1141 had takeoff configuration warnings that were to alert the

pilots if the aircraft was not properly configured, in both cases the automation failed to

detect the absence of flaps and/or slats.

Automation does not guarantee safety for aircraft configurations (Degani and

Weiner, 1993; Palmer and Degani, 1991). The lack of complexity of most general aviation

aircraft preclude the benefits automation provides. Limited electronics and lack of take­

off configuration warnings require the general aviation pilot to rely on the actual execution

of the checklist and personal experience in the aircraft to ensure that the aircraft is

properly configured. Most general aviation aircraft do not have sophisticated systems,

although many do have simple systems. An example of this would be a horn which sounds

if an aircraft with retractable gear approaches a slow (landing) airspeed without the gear

extended. Despite this, many aircraft land gear-up every year. What is not known is how

8

many accidents are prevented each year by these types of aircraft configuration warning

systems.

Checklist design has implications as to whether it will be completed accurately. If

it is designed so that the pilots consider it cumbersome or an obstacle to complete, it may

be regarded as a nuisance task and lead to short-cuts or individual procedures (Nagano,

1975; FAA, 1995). If the checklists are excessively lengthy, pilots may inadvertently skip

over items (Degani and Weiner, 1993). This short-cutting of checklists, whether

purposefully or inadvertent, may render it ineffective.

The order of items on the checklist may also play a critical role in how well the

checklist is accomplished. "This order of items is the only indicator of the pilot's point of

progress in the checklist" (Degani and Weiner, 1993, p. 355). One must consider the

question that a predictable order of items on the checklist may lead to potential problems

that may be associated with a task that is done repeatedly. Would problems such as

complacency and reduced sensitivity to the mechanism render the checklist ineffective?

When taking off or landing an airplane, although the task may be the same, flight

conditions are unique. No two takeoffs or landings are ever the same due to external

variables, such as weather, etc. Therefore, the pilot's attention must be at its highest

during the takeoff and landing parts of the flight. If a method that would allow for the

same degree of attention could be applied to checklist use, then perhaps vigilance in

checklist operation would be increased.

As with many complex machines, certain tasks must be sequenced in a logical

order according to the activation of systems. Other systems may be checked in a less

9

structured order because each item may not have direct bearing over the item that

precedes or follows it (Degani and Weiner, 1993).

The order of checklist items generally follow a pattern of eye and motor movement

in the cockpit. This is referred to as flow pattern (Degani and Weiner, 1993).

If the established flow patterns are not logical and the checklist itself correct and consistent with procedures prescribed in related manuals, the probability is very high that the crew may, when pressed for time, revert to their own methods, cut corners, omit items, or even worse, ignore the checklist entirely (FAA, 1995, p. 18).

Degani and Weiner (1993) recommend that very critical checklist tasks be

accomplished first. They reason that the captain will usually call for the checklist when the

workload is low and consequently the probability of completion is high. Also, the

probability of accomplishing subsequent tasks diminishes as time progresses. Degani and

Weiner (1993) noted that some pilots have developed their own personal check of certain

"killer items" prior to takeoff. If a checklist is properly designed and executed, then there

would be no need for a personalized procedure.

Checklist initiation and interruption

When a checklist is to be initiation is another important factor in its design and use,

and may be dependent entirely on the captain. Because of external pressures, such as

radio communications, ramp personnel, passengers, or other crew members, the checklist

may be initiated at an inappropriate time. The NTSB surmised from the cockpit voice

recorder of Northwest flight 255 that the checklist was not completed due to the first

10

officer being distracted at the last minute because of a runway change and the captain's

passive involvement with checklist initiation (Sumwalt, 1994).

Checklist interruption and resumption also has an influence in the quality of the

checklist procedures. A study conducted by Linde and Goguen (1987) found that when

the checklist is interrupted by ground personnel, air traffic control or other crew members,

it may not be resumed at the proper place or totally abandoned. Use of an explicit hold,

such as "hold at name of checklist item " by the initiator of the checklist significantly

reduces the possibility of an incomplete checklist. "If this is done, the load on memory

may not be as great, since placing such a hold makes a social acknowledgment of the fact

that the checklist has been interrupted" (Linde and Goguen, 1987, p.4).

Single pilot operations do not have the benefit of another crewmember to remind

them of checklist interruption. Again, the single pilot has to remember to complete the

checklist if interrupted. If an erroneous cockpit indication is detected during the checklist

process which requires the pilot to stop and determine the nature of the problem, the time

to complete the checklist will increase. Although an interruption of this nature may stop

the flow of the checklist procedure, the additional time may increase the probability of

error detection and correct checklist completion. This research will explore this element

of checklist completion.

11

Psychological Factors

There are several psychological and physiological factors that have implications as

to whether the checklist is performed in an adequate manner. Fatigue, stress, pilot

workload and distractions all have bearing on the human element in checklist performance.

Fatigue can reduce pilot performance because of behavioral changes. One compensatory measure a fatigued person may revert to is one of lead shedding or workload reduction. A fatigued individual tends to rank tasks to be accomplished according to their perceived importance and sheds or deletes those of a lower priority. The individual may also refuse to accept new tasks or inputs or may devote less time to each of the present tasks. This can directly impact the successful completion of a checklist tasking (FAA, 1995, p.9).

Personal stress unrelated to flying may still manifest itself in pilot behavior by

markedly decreasing alertness and performance. Emotionally upsetting events such as a

death of a family member or friend, arguments, financial pressure or job security concerns

may alter the pilot's ability to concentrate on all the various aspects involved with aircraft

operations.

Workload and task management is another area of concern to the pilot. The use of

the checklist generally occurs in phases of high workload. Based on a review of NTSB

accident reports and ASRS incident reports, the flight crew is most vulnerable to checklist

error in flight operations prior to take-off (FAA, 1995). Other research indicated that

90% of all time-related human errors occurred in the pre-flight or taxi-out phase of

operations (McElhatton and Drew, 1993). For this reason, this phase of operation is most

salient to this research.

12

Before taxi, pilots are involved with the numerous activities in preparing for a

flight. These activities include: gathering and disseminating weather information,

conducting weight and balance computations, aircraft fueling, baggage loading, passenger

loading, and briefings. Pilots are also confronted with push-back crews, dispatchers, flight

attendants, and mechanics, all of whom may make demands on the crew in the pre-taxi

phase. These interruptions may be non-linear in nature; that is, they may occur at non­

specific times during the pre-taxi phase. It may be the pilot's own judgment as to

prioritize these issues and attend to them (McElhatton and Drew, 1993).

In contrast, the approach and landing phase is usually less hectic, more predictable

and linear in nature.

The task of landing associated with the arrival sequence to an airport is generally spread over a longer time period...and the cues for initiating the checklist are normally more pronounced, e.g., descending out of cruise altitude, perform "DESCENT" checklist; at or near the final approach fix complete the "BEFORE LANDING" checks (FAA, 1995, p. 10).

This is also true in general aviation flying. While many of the responsibilities of the

airlines are not present, the interruptions that can occur prior to taxi can be just as

disruptive, especially if the pilot does not fly often or is inexperienced.

Human Information Processing

The cognitive portion of the checklist task involves the processing of information.

During the checklist procedure, the pilot is to identify potentially hazardous situations.

More specifically, the pilot must go through a series of psychological processes to

13

determine if the aircraft is configured properly. These include: attending to instrument

and outside scanning, identifying proper aircraft configurations, and relying on long term

memory for recall of required configurations. The pilot must also take into account

previous experience with the route of flight, weather, runway, aircraft performance, and

air traffic control. How this is achieved can be explored using theories in classic vigilance

research.

Memory

Memory plays an important part in information processing. Unaided working

memory has been shown to have the capacity of retaining approximately seven (plus or

minus two) unrelated items. "Unless actively rehearsed, or aided by some external form of

reminder or memory jogger, information contained in the working memory will generally

be forgotten in 10 to 20 seconds" (Miller, 1956; Wickens, 1988; Flach, 1988 in FAA,

1995, p. 12).

Problems that arise from the memory process are the result of several potential

information encoding errors. Information encoding errors are failures to properly encode

the new information in such a way that it can be properly recalled. Encoding errors are

the result of incorrect mental models. Mental models are formed when a task is repeated

so often that it becomes rote and set expectations result.

When a certain task is performed in the same manner, operators become experienced with the task. In a sense, they actually create a "mental model" of the task. With experience, the shape of the model becomes more rigid, resulting in faster information processing, ability to divide attention, and consequently leading to a reduction in workload (Degani and Weiner, 1990, p.3).

14

Wrong mental models may occur as a result of insufficient environmental

information or experience, incorrect expectations of the pilot, or an incomplete mental

model (Mangold and Eldredge, 1993). Incorrect mental models may result in pilots

believing that a checklist item is completed when, in fact, they only "saw" what they

expected to see. "...(T)his (mental) model may adjust, or sometimes override, the

perception of physical stimuli coming from the receptors and bias the brain" (Degani and

Weiner, 1990, p.3). This is supported by Aviation Safety Reporting System data and may

have been the situation for Delta flight 1141 in which the checklist call for flaps was made,

but the actions were not performed (NTSB, 1989, in Degani and Weiner, 1993). 30

Davies and Parasuraman (1982) suggest that working memory may play a

significant role in the cognitive demands of successive vigilance tasks. They identify

successive discriminations as those in which recall of a standard be compared with the

signal. In the example of pilots preparing the cockpit prior to checklist use, the entire

cockpit can be considered a signal. Successive discrimination may be learned in pilots

with considerable experience in a particular aircraft or when using one checklist

exclusively. A glance at the cockpit configuration may provide pilots with sufficient

information with regard to cockpit layout and they may configure (reconfigure) the

aircraft without the necessity of the checklist.

Simultaneous discriminations are those in which judgments about the signal are

made without regard to a remembered standard, i.e., a checklist "do-list" or a checklist in

which the items are randomized. Simultaneous discrimination may be the preferred

15

method for pilots who use the checklist as a "do-list" without first configuring the aircraft

from memory out of habit, necessity, or preference.

Successive discrimination (the visual cockpit check) may have more of a drain on

the subjects' resources than simultaneous discrimination (use of the checklist). If this is

the case, there may be less cognitive resources with successive discrimination and may

cause expectancy in vigilance to be hampered (Davies and Tune, 1969, in Dittmar et al.,

1986). Conversely, simultaneous discrimination, i.e., judgments not compared to a

remembered standard, would not hamper working memory and therefore may not

adversely affect vigilance. Simultaneous discrimination would not require the pilot to

memorize a checklist or the exact aircraft configuration. Consequently, a checklist could

vary each time and it may not have an influence on the accuracy of the configuration.

Which type of discrimination is being developed, and which is more desirable for

the single pilot operation? Does the type of discrimination have bearing on time to

complete the checklist? If a pilot uses standard checklist exclusively, does successive

discrimination take place? If so. what is the implication in checklist use? These questions

will be explored by this research.

Expectancy Theory

Dittmar, Zeileniewski, Dember and Warm (1986) conducted a study to determine

if signal regularity had an effect upon perceptual sensitivity. Expectancy Theory states

that a person will form a schedule of expectancy of a signal after experience has shown

approximately when a signal will arise. The subjects will consequently modify their

16

attention and their vigilance remains high. This theory also states that vigilance drops

immediately after a signal. Dittmar et al. (1986) suggests that Expectancy Theory may

play a significant role in the vigilance of a subjects.

The pilot may form schedule of expectancy with regard to particular checklist

items that trigger a major motor actions. In the case a pre-departure checklist for a light

twin aircraft, an example of this would be moving the hand to the ignition key. If

Expectancy Theory holds true for pilots that use the same checklist each time, one might

suspect that vigilance would alternately be high and low. While attending to each

individual task, vigilance would be high because each item on the checklist can be

considered a signal. After each item is accomplished on the checklist, the vigilance for

that particular item drops. Additionally, if the pilot does not have a second check or the

benefit of automation, an item that is improperly set may not be discovered until a problem

develops due the resulting low vigilance after each checklist item (signal) is completed

(detected).

A checklist that would allow for continued high vigilance would be desirable. If

the checklist varies each time, then the pilot would not know exactly when the particular

item would be on the list. Using the example above, the pilot completes the checklist in

anticipation of the item that calls for the major motor movement (key the ignition switch).

This may result in increased vigilance throughout the entire procedure.

17

Automatic and controlled processing

Fisk and Schneider (1981) suggest that vigilance and associated performance

decrements are the result of the type of cognitive processing involved in tasks that require

sustained attention. Schneider and Shiffrin (1977) refer to these processes as automatic

and control processing, which are quantitatively and qualitatively different.

Fisk (1985) suggests that the consistency between signals and the subject's

practice has an effect on vigilance. Fisk and Schneider (1981) concluded that the mode of

information processing, automatic or controlled, has significant bearings on the ability to

maintain attention. This leads to two questions: which type of mental processing is

involved with pilot checklist procedure and what is required to adequately train people to

perform checklists better?

Automatic processes are fast, parallel and fairly effortless. They allow

performance of well-developed, skilled behavior, are not limited to short-term memory

capacity and they require extensive training to develop. Automatic processes develop in

situations where subjects consistently respond to stimuli (i.e., they are always attending to

and are never ignoring a stimulus when it occurs). Automatic processing, although takes

significant training and exposure to the task, as well as refresher training, have

significantly better vigilance performance than those tasks that involve control processing.

Control processes are comparatively slow, serial, effortful and capacity limited.

Controlled processes deal with novel or inconsistent information and require little training

to develop.

18

Which type of mental processing is preferred for pilots during checklist operation?

Degani and Weiner (1990) describe checklist use as follows:

[T]he combined effect of expectation, experience and pattern analyzing mechanism is a double edge sword. On one side, this ability makes the user flexible and faster in responding to multiple conditions. On the other side, it can lead the operator to make a disastrous mistake just because part of the information which was collected quickly or without sufficient attention appeared to match the expected condition (p. 40).

The pilot conducting the checklist must first read the item on the sheet, search for

the item in the cockpit, verify its actual position (state), determine whether the item needs

to be changed, change the status of the item if appropriate, then check it again to ensure

that it was done properly. If the checklist in use has the same items in the same order,

then this task fulfills part of the requirements for automatic processing. However, some of

the items identified as controlled processes are also required. Each flight may have unique

characteristics, some of which are: the pilot may not be very familiar with the aircraft or

route of flight, the pilot may not have current flight time, or the checklist may be new to a

familiar aircraft. The pilot must spend additional time to determine if the checklist is being

completed properly.

Fisk (1985) supports the work of Fisk and Schneider (1981) and Schneider and

Shiffrin (1977) in which the relationship of automatic or controlled processing has an

influence on target detection rate. Controlled processing may become automatic when

given consistent and extensive practice. However, the practice must be consistently

mapped (CM).

19

"[T]he individuals [must] make the same overt (or covert) response to stimuli (or class of stimuli). If the individuals receive varied mapping (VM) training - i.e., a given stimulus requires responses that change across time - automatic processing will not develop and performance will not substantially change with practice" (P-15).

Checklists that are the never changing may allow for automatic processing of the

procedure. Training of this type would be considered as consistently mapped (CM) and

time for checklist completion would decrease. If the checklist varied by sequence, varied

mapped (VM) training, then automatic processing would not take place and therefore the

time for checklist completion would not decrease.

During any checklist that is conducted while on the ground, the pilot is able to

look as long as necessary to determine if the checklist item is properly configured.

Because the pilot has this ability to take as much time as necessary during the pre-

departure checklist, the task on the ground differs from a checklist that is conducted while

under flight. In flight, the pilot's attention must be divided to include the search task while

flying. In theory, the pilot while on the ground has unlimited time and opportunity to

discriminate each checklist item and eliminate the errors without time or speed constraints.

If this is always the case and time is theoretically unlimited, why are checklist errors still

being made? Two of these reasons may be complacency and the predisposition to hurry.

Checklist complacency

Hawkins (1988) suggests that the greatest enemy of error-free discipline checklist

use is attitude of the pilots involved. This attitude trickles down from airline management

20

to the chief pilot and ultimately to the flight crews. However, in most airline cultures, the

emphasis is on safety and strict checklist adherence.

Airline pilots are required to fly many flight segments (each takeoff and landing

segment is referred to as a "leg") during the course of one day. Pilots generally work

several days in a row flying a series of flight legs. This is referred to as a trip. This

requires performing checklists repeatedly, often 3-10 complete (pre-departure to stopping

engines) checklists per day and as many as 10-30 complete checklist per trip. Commuter

pilots may have as many as 16 flight legs per day. The requirement to conduct a full

checklist procedure prior to each flight may be viewed as a nuisance (Degani and Weiner,

1990). Pilots with more experience may face the prospect of checklist procedure as

cumbersome to the flight rather than an integral part of the flight itself (Degani, 1993).

This may lead some pilots to abbreviate the checklist or not use it to back-up their

configuration of the aircraft.

Olcott (1991) describes checklists as "tools of the professional" that protect the

inexperienced pilot against lack of familiarity and guards more experienced pilots from

complacency. General aviation pilots are at the disadvantage of not having the benefits of

another cockpit crewmember to guard against abbreviating the checklist.

Airline philosophy and culture advocates safety, especially with the ever increasing

acceptance of CRM training which encourages active crew participation in flight deck

procedures. The single pilot, operating in the general aviation environment, does not

always have the benefit of crew and company support. Other than mandatory biennial

flight reviews, general aviation pilots are not required to undergo significant training

21

programs once they leave the flight training environment. The typical general aviation

pilot may fly with a flight instructor for only one hour every two years. The pilot's own

individual habits may have evolved from strict checklist discipline while in a training

environment to individualized procedures that may exclude strict cockpit discipline.

Additionally, familiarity with only one type of aircraft may lead pilots to expect checklist

items were completed when in fact they were not (Hawkins, 1988).

Visual, tactile and motor skills become involved in the verification process. The

combination of motor movement with mental sequencing process aid the interpretation of

the checklist item (Degani and Weiner, 1993). Because of familiarity with the checklist,

the pilot responding may do so without actual verification of the item. The motor skill

involved with the checklist may become so ingrained that the physical response can

become as automatic as the non-specific verbal response (e.g., "Checked", "Set"). This

may render the physical motion ineffective if the pilot is not mentally engaged in the task.

For example, the captain's call for the checklist may cause the co-pilot to physically touch

the checklist card, but not remove it from its holder and read the item. Instead, the co­

pilot will perform the checklist from memory. Degani and Weiner (1990) observed this

behavior during both day and night observations of flight crews. If the pilot reading the

checklist does not look up and verify the response of the other pilot, the redundancy of the

check is eliminated.

It is this habit of rote response that investigators believe contributed to the Delta

1141 accident. During the analysis of this accident, the NTSB investigation measured the

time delay between the second officer's challenge ("flaps") and the first officer's reply

22

("Fifteen , fifteen, green light") as recorded in the cockpit voice recorder (CVR). They

reported that, "...the time between the checklist challenge and responses was less than one

second, with little time to accomplish actions required to satisfy the proper response

"(NTSB, 1989 p. 61, inDegani and Weiner, 1990, p. 39).

Hurry-up Syndrome

The predisposition to hurry can contribute significantly to degradation in human

behavior. This is known as the "Hurry-up syndrome" and is defined as "any situation

where a pilot's human performance is degraded by a perceived or actual need to 'hurry' or

'rush' tasks or duties for any reason" (McElhatton and Drew, 1993, p. 1).

A study of ASRS incidents by FAR 121 and 135 operations that contained the

word "hurry" or "rush" was conducted by McElhatton and Drew (1993). They found that

the vast majority of time pressure errors occurred in the first two operational phases of

flight, that is, pre-flight and taxi-out. The results of these errors manifested themselves in

either the phase where the human error occurred or the phase immediately following.

They also note that "errors are less likely to be detected in a high-workload, time

compressed flight phase rather than in a low workload flight phase encountered some time

after departure" (McElhatton and Drew, 1993, p. 5).

23

Summary

Aviation accidents have led to the further study of the human factors implication in

checklist design and usage. Research indicates that 90% of all time-related human errors

occurred in the pre-flight or taxi-out phase of operations. For this reason, this phase of

operation is most salient to this research.

The two basic philosophies of checklist design and use are the "challenge-and-

response" method that is primarily used in airline operations, and the "do-list" which is

most common in general aviation operations. The most frequently used checklist in all

aviation operations consists of a printed card with two columns.

The order of items on the checklist may play a critical role in how well the

checklist is accomplished. A pilot who uses the same checklist repeatedly may not pay

sufficient attention to the task. This may lead to rote operation of the checklist procedure

and, consequently, complacency, which may affect the pilot's mental model of the

situation. Pilots may expect checklist items were completed when, in fact, they were not.

A checklist that varied each time may prevent rote execution of the checklist procedure

and prevent it from becoming routine. More cognitive resources may be dedicated to

checklists attention which may result in longer time to execute the checklist. Thus, the

pilot may be provided with more time to carefully analyze the aircraft configuration and

detect potential errors.

24

Hypothesis

Pilots in training may be so accustom to using a fixed, unchanging checklist that

unchallenged repetition may lead to cockpit complacency indicated by procedural errors.

It is hypothesized that a checklist that had a randomized order would result in fewer errors

than an unchanging checklist. Additionally, the randomized order would increase the time

it takes the pilot to perform the checklist.

Method

Subjects

The subjects for the study were Bridgewater State College students who held at

least a student pilot certificate, had completed at least one solo cross country, and logged

a minimum of three hours in the ATC-810 multi-engine simulator. A request for

volunteers was made through the aviation classes. They each filled out a form that

contained their name, phone number, total flight hours, total time in the ACT-810

simulator and their availability for the study. The names, phone numbers and availability

times were required to efficiently schedule their times for the study. All of the subjects

were required to have at least three hours logged in the simulator within the past four

months.

Because the preliminary request for volunteers indicated a wide diversity of flight

hours, licenses, ratings, and hours in the simulator, it was suspected that this varied

experience level may have a significant impact on the outcome of this study, the

simulator is primarily used by the college to teach navigation and instrument procedures.

Although simulator hours may affect the ease with which the subject flies the simulator, it

may not have as much bearing on actual flight performance as total flight hours.

Therefore, the subjects were categorized by total flight hours and a stratified random

sample was generated. Stratification is shown in Table 1.

25

26

Table 1. Pilot hours stratification

Subjects

4

3

11

2

Flight hours

<100

100-200

200-400

>400

There was one student pilot, five private pilots, six instrument-rated private pilots,

seven commercial/instrument pilots and one flight instructor. Each of these groups were

randomly divided. Half were selected as control subjects and the other half were subjected

to experimental conditions. Overall, one female and nineteen males participated in the

study.

Although a few of the subjects were multi-engine rated, most were not. The

subjects were informed that during the study they would not be subject to any in-flight

engine failures or emergencies.

Equipment

The ATC-810 simulator and the standard ATC-810 checklist at Bridgewater State

College were used in this study (see Appendix A). The ATC-810 simulates a single-pilot,

light twin-engine airplane, similar to a Piper Navajo or the twin-Cessna 340, 402, 414

series. It is equipped with standard cockpit displays for altitude, airspeed, vertical speed

and turn coordinator, a flight director, and horizontal situation indicator (HSI).

Navigation and communication radios are standard dual nav/com type with two very high

27

frequency omi-directional range radios (VORs, one of which is displayed on the HSI), two

standard communications radios, one automatic direction finder/radio magnetic indicator

(ADF/RMI) unit and a distance measuring equipment (DME) indicator.

The checklists were broken down into five groups of items: PREFLIGHT,

STARTING, TAXI CHECK, TAKE-OFF/CLIMB, AND LEVEL OFF/CRUISE. The

pre-landing, approach, landing and stopping engines procedures were not included in the

study because the trials were terminated when the subject reached the predetermined

checkpoint on their flight.

The "Starting Engines" checklist was of primary focus in this study because most

of the errors committed in a checklist procedure occur during the execution of the

preflight checklist (McEllhatton and Drew, 1993). For the varied checklist used in the

experimental group, randomization of the sequence of the items in the "Starting Engines"

checklist was accomplished without adversely affecting the outcome of the checklist

procedure. In a few, select instances, the items on the checklist could not be logically or

appropriately varied. Those sequences were: the turning on the simulator sequence and

the engine start sequence. These two sequences were at the very beginning and the ending

portions of the "STARTING" checklist respectively.

There were twenty-five items on the "STARTING" checklist. Items four through

nineteen were presented in randomized order. Those items were written on a piece of

paper and placed in a hat. The papers were then drawn for order of their new sequence

for the varied checklist. There were eight different checklists generated in this manner. In

order to give the appearance of uniformity, there were no indications on the front of the

28

checklist (the only side with written items) that would indicate which checklist was being

used. The checklists were numbered on the back in the upper right-hand corner in pencil

that indicated which varied checklist it was. There were also eight control checklist

generated. For each trial of the control groups, the subjects received a different copy of

the same checklist. The checklists used in both groups can be found in Appendix A.

Procedure

The subjects were informed that intent of the research was to determine their

accuracy throughout all aspects of flight. It was intended that the subject be unaware of

the specific intent of the research so as to not to unduly influence them to pay special

attention to the checklist. Each subject participated in the study independently; there were

no concurrent trials.

Prior to participating in the study, each subject completed an informed consent

form which stated that participation was voluntary and the results would be anonymous.

This form can be found in Appendix B.

The subject was handed a clipboard that held the trip sheet and the checklist and

was instructed not to write on the checklist. The researcher then explained the procedure

that would be followed during the study.

Each subject was involved in one two hour session which involved eight trials.

Group I, the control group used the same checklist for each trial. Group II, the

experimental group, used the varied checklist for each trial. For each trial the subjects

were instructed to perform a pre-departure checklist and proceed to the first checkpoint of

29

a cross country flight in the simulator. They were each given an instruction sheet that

contained departure instructions, as well as headings and courses to fly. They were also

instructed to record the time on take-off and reaching the checkpoint. All frequencies

were given to the subject and each departure was reviewed with the subject by the

researcher at the conclusion of the "TAXI CHECK". Upon reaching the first checkpoint,

the simulator was frozen which indicated the end of that particular trial. The subject was

then asked to leave the room for twenty seconds and the simulator was then reset. During

that time, the checklists were exchanged. The control group received a new copy of the

standard checklist while the experimental group received a new varied checklist. These

were reattached to the clipboard in the same manner as the subject had been using. The

subjects were asked to return to the simulator when the twenty seconds had expired. This

process was repeated for all trials. The trip sheets for the subject's flights can be found in

Appendix C.

Of the eight trials, the first six contained no unusual events. The remaining two

trials, 7 and 8, each contained two random faults in the cockpit that would be discovered

if the checklist was properly used. This is represented in Table 2.

The faults were randomly assigned from a list of faults determined by expert

opinion to be of a typical nature for a light twin aircraft. The experts consulted were the

Bridgewater State College flight simulator lab instructor and the coordinator of the

College's Aviation Science program. These faults were set in the cockpit as appropriate

when the subject was out of the room.

30

Table 2. List of faults during the trials.

Trial 1

2

3

4

5

6

7

8

Faults none

none

none

none

none

none

2 random

2 random

An error was said to have occurred if: the fault was not found, the fault was found

but not acted upon, or the fault was found but acted upon incorrectly and the subject

elected to take-off. The data was categorized as being "fault found" or "fault missed". If

the fault was found then no error will have occurred.

The period from the time the subject turned the key switch (item 2 on both

checklists) until the point where the subject turned on the radio (item 23) was recorded.

These points were chosen because there was a very definable motor movement by the

subject that would indicate exactly where the subject was on the checklist. This data was

recorded (see Appendix E).

It was anticipated that the subjects could commit errors that were not in the

experimental design, however, no such errors were observed.

Results

Errors

Total errors committed in the control group were four errors in trial seven and two

errors in trial eight, thus making a total error rate of six for the control group. The total

errors committed by the experimental group was one error in trial seven and five errors in

trial eight, thus making a total error rate for the experimental group of six. This is shown

in Table 3.

Table 3. Total error rate for both groups.

Control Group

Experimental Group

Trial 7

4

1

Trial 8

2

5

Total

6

6

Thus, the research hypothesis that stated that there would be a difference in error rate

between the two groups is not supported. A chi square test was performed to determine if

there was a significant difference between groups as a function of trials. The results of the

chi square test indicated that there was no difference: observed x2=3.08 critical x2=3.84.

31

32

Effect of checklist design on time

The times of both groups to perform the checklist is shown in Figure 1.

Figure 1. Graph of the times of the control and experimental groups.

A two-factor mixed design, repeated measures on one factor analysis of variance

was conducted to determine if there was a statistical difference between the times of the

control group and the experimental group to complete the checklist for trials one through

six. This is shown in Table 4.

The times, means, and errors of the subjects can be found in Appendix F. Also,

the subject's total flight hours and hours in the ATC-810 can be found in Appendix F.

33

Table 4. Effect of checklist design on time.

Source

Total

Between subjects

Conditions

Errorb

Within

Trials

Trials x conditions

Errorw

SS

155,641.99

101,245

20,046

81,199

54,396

20,358.35

4,762.30

29,275.35

df

119

19

1

18

100

5

5

90

ms

20,046

4,511

4,071.67

952.46

325.28

F

4.44

12.51

2.92

P

<05

<05

<n.s.

Therefore, it is concluded that the results indicate a significant difference for

conditions (checklist) and trials (learning): Critical F(1;18) 4.41; Fc(5,90) = F5>60,4.76.

There was no significant interaction between the groups.

The ANOVA also revealed a significant difference in time for trials one through

six: Critical F=4.44, observed F=12.51. This indicated that learning took place over trials

one through six.

A three-factor mixed design-repeated measures on two-factors analysis of variance

was performed to determine if there was a significant difference on time between the two

groups on trials five and six (no faults) versus trials seven and eight (introduced faults) at

34

the p_ =.05 level of significance. There was no significant difference found. The results are

shown in Table 5.

Table 5. A three-factor mixed design-repeated measures on two-factors analysis of variance comparing time to perform the checklist between trials five and six versus seven and eight.

Source

Total

Between subjects

Groups

Errorb

Within subjects

Normal-Fault

Trials

G x F

G x T

F x T

G x F x T

Errori

Error2

Error3

SS

77,862.80

49,677.30

4,530.05

45,147.25

73,332.75

9,856.80

51.20

130.05

84.05

12.80

110.45

9,495.65

4301.25

49,290.50

df

79

19

1

18

60

18

18

18

ms

4,530.05

2,530.05

9,856.80

51.20

130.05

84.05

12.80

110.45

527.536

238.958

2,738.361

F

1.806

1.038

.214

.246

.226

.0046

.040

Fc(.o5)=4.41

n.s.

n.s.

n.s.

n.s.

n.s.

n.s.

n.s.

Discussion

The results of this study do not support the research hypothesis that a checklist

which covered the required items but varied by sequence would reduce procedural errors.

There was no difference in the total error rate between groups. There was a statistical

difference in time to complete the checklist between groups and over trials (1-6). There

was no statistical significance between groups after a fault was introduced in trial seven

and eight.

Observations

Both the variable checklist and the standard checklist provided the required

foundation for verifying aircraft configuration. The subjects seemed to be well aware of

the importance of the checklist and had a positive attitude toward the procedure. None of

the subjects attempted to perform the flight without using the checklist.

The subjects had no questions as to how to use the checklist. Their simulator and

flight training experience involved use of a checklist card. As expected, nearly all of the

subjects performed the checklist as a "do-list." The training environment for single pilot

operations encourages this type of checklist use. Some of the subjects read aloud both the

left and right columns (challenge and response) while they performed the procedure. One

35

36

subject, however, used the airline approach to checklist use; a measure of redundancy to

back-up an initial configuration. A review of this individual's experience revealed that his

flight time was the second highest (over 800 flight hours), and his flight experience

extremely varied. It included may different types of aircraft simulator hours, including B-

727 and B-747 simulator time. Interestingly, his time to complete the checklist was the

longest of all the subjects.

Because the ATC-810 simulated a light-twin aircraft, it also had some simple

automation. For example, when the landing gear fault was introduced in the cockpit by

having one of the three green (gear safe) lights extinguished, the warning horn sounded if

the throttles were set at the full idle detent. Some of the subjects, upon hearing the horn,

did not fully realize what had triggered it. It took some time before they noticed that one

of the landing gear bulbs was extinguished. It was then that they realized that their check

of the gear position lights at the beginning of the checklist was deficient.

Only a few of the subjects in the experimental group noticed that each checklist

was different. This may have occurred because the flow pattern changed with each

checklist. It is uncertain if the all the subjects in the experimental group noticed the

randomized order of the checklist items. Of the subjects that did notice a difference, it is

uncertain whether it caused them pay special attention to the checklist. Post-test

interviews may have provided some insight into what the subjects observed. No post trial

interviews were performed in this study.

It was also observed that many of the subjects touched items that were on the

checklist without actually looking at the item (e.g., LANDING GEAR...DOWN). This

37

was observed in both groups. In some instances, an error resulted, in other instances, the

item was later verified visually. Although a motor movement is recommended by Degani

and Weiner (1990), they also recommend that this movement be accompanied by a visual

check at the same time.

The specific faults that were randomly set and those missed by the subjects are

shown in Table 6.

Table 6. List of randomly set faults.

Faults 1

2

3

4

5

6

7

8

Gear

Fuel pump

Roll trim

Cowl flaps

Oil pressure

Fuel selector

Circuit breaker

Flaps

Set 11

10

13

8

6

12

11

9

Missed 2

1

3

0

1

1

4

0

It is interesting to note that the faults most commonly missed in this study were the

tripped circuit breaker fault and the roll trim. The circuit breaker that was tripped was the

right engine oil temperature gauge. The subjects that were randomly assigned the circuit

breaker fault had a resulting fault of an inoperative oil temperature gauge. This gave the

subjects two clues of an existing fault: the white band of a tripped circuit breaker and the

corresponding inoperative electrical gauge. Failure to detect both cockpit indications had

to occur in order for the fault to be counted.

38

One possibility why the subjects missed both of these clues is that the subject's

flight or simulator training may not stress verification of the circuit breakers. The circuit

breaker in the simulator is different from the type that is most commonly used in training

aircraft. The simulator circuit breaker more closely resembles one that can be manually

tripped and reset, such as those found in more sophisticated aircraft. In most single engine

aircraft, the circuit breaker can only be reset after it trips due to an electrical problem.

This may have resulted in the subjects' failure to recognize the tripped circuit breaker.

The circuit breakers were located directly in front of the pilot. The right engine oil

temperature gauge that became inoperative as a result of the tripped circuit breaker was

located near the top, right hand corner of the instrument panel. Even thought the checklist

called for verification of oil temperature after engine start, the inoperative gauge was not

detected. There are several possible explanations for this. The subjects may not have

checked the gauge, may not have looked at the correct gauge, or incorrectly interpreted

the gauge. If the subject did not interpret the gauge correctly, then the subjects may have

experienced an incorrect mental model. The subject may have perceived that the gauge

was operating properly, when, in fact, it was inoperative.

The roll trim was the other most commonly missed fault. There are several

possible explanations for this. Roll trim is rarely present in basic training aircraft. The

subjects may have only been exposed to it in the simulator. The attention that the subjects

gave to the roll trim as a result may have been high, due to it being novel and unusual, or

it may have been ignored because the subjects had little practical experience with it in

training aircraft.

39

The failure to detect the roll trim fault may also have been a result of incorrect

information processing. The subject may have misinterpreted the visual stimulus of the

gauge and "saw" only what they expected to see. It can also be theorized that these

particular faults occur most frequently because these items are infrequently manipulated by

the pilot. The subjects that missed the roll trim during the checklist became acutely aware

of the incorrect setting after lift off when the aircraft began to roll sharply.

While the possibility existed for the checklist to be abandoned or resumed at an

improper place when interrupted by a fault, this was not observed for any of the subjects.

It was also noted that some of the subjects put their thumb at the point on the checklist

where the interruption took place so they could resume without skipping any items.

The introduction of most faults caused the subjects to stop and spend whatever

time necessary to determine the nature of the fault. However, in two instances, the

subjects experienced no checklist interruption because they did not discover the faults.

The physiological and psychological status of the subjects was not inventoried.

Therefore it cannot be determined if fatigue, stress, or major emotional events had any

negative bearing on the subject's performance.

The time of each of the eight trials was brief, approximately fifteen minutes.

Hence, a checklist was performed eight times in the two hour test trial. Although learning

would have taken place outside the controlled environment, the compressed time frame

may have hastened familiarity with the checklist. Learning took place as indicated by a

steady decrease in time in both groups from trials one through six, with a learning plateau

attained by trials five and six.

40

Analysis

Even though the time to complete the checklist was not statistically different

between the two groups, it is interesting to note that the mean times for each of the trials

of the experimental group was less than those of the control group. This was unexpected.

It was expected that the pilots in the control might develop automatic processing and

result in faster completion time of the checklist. It is unclear which type of discrimination,

successive or simultaneous, if any, was developed. It may be that the number of trials was

insufficient for the subjects to develop automatic processing. More trials may have had a

greater impact on the subject's ability to develop automatic processing of the checklist

procedure.

It was anticipated that repetition of an unchanging checklist would result in an

overall shorter time than the variable checklist. Because the sample size was small (n=20),

this result may be due to sampling error and/or individual differences of the subjects.

There may have been insufficient subjects and insufficient number of trials to accurately

test the research hypothesis. Secondly, the subjectis total flight hours and hours in the

ATC-810 varied widely: the subjects were stratified based on total flight time. Further

analysis may determine if ATC-810 simulator hours had a bearing on the observed

differences.

It is also possible that the subjects had too little overall flight experience to become

complacent. They were all involved in a flight training environment and most likely to be

conscientious in the performance of the checklist. The flight training environment at the

college stresses the importance of safety which may have had influence on the research

41

results. It cannot be determined if additional flight experience would have increased or

decreased the overall error rate or how it would have affected the overall time to complete

the checklist.

Another plausible explanation for this unexpected result could be the measure

itself. The number of items that were involved in the randomization was fourteen. Even if

sampling error did not take place, this checklist may not have been complex enough to

reveal a difference between fixed and varied checklists. Perhaps a more complex single-

pilot aircraft would have a checklist of correspondingly greater detail that would require

greater vigilance.

A variable that was not explored in this research was that of time pressures or

constraints. The time/accuracy trade-off may play a role in error detection. In real world

flight scenario, pilots are constantly under time pressure, whether overtly by airline

schedules, weather, flight/duty time limitations, ramp or taxiway congestion, or covertly

by passengers or the pilot's own desire to get into the air. Perhaps the error rate would

have increased if the subjects were given a "clearance void time" to simulate a time

pressure and have resulted in a predisposition to hurry. Further research into this area is

recommended.

Conclusion

Aircraft checklist design and use has received considerable attention in recent years

as a result of several unfortunate accidents due to checklist misuse. Because of lack of

sophisticated recording devices, it is unclear how many general aviation accidents are a

result of checklist misuse or non-use.

The results of this study indicate that the pilots in this training environment commit

the same amount of errors regardless of whether the items on a checklist are in the same

order or vary each time. Therefore, it is inconclusive whether a randomized checklist

would be more advantageous for the general aviation pilot.

Recommendations for further research

It is recommended that further research into this area be conducted using a far

greater number of trials to increase the likelihood of automatic processing taking place. It

would also be interesting to investigate how many trials and checklist procedures would

be required to determine where complacency may override checklist discipline.

Another area for further investigation would be to determine how many variable

checklists would be necessary to provide the level of uncertainty or limited expectation

42

43

that would cause an increase in time and hopefully, accuracy. However, any checklist, if

used frequently enough, will be at risk of becoming routine.

It is also recommended that further research investigate the possibility of time

pressures on the checklist procedure to determine if a relationship exits between time and

error rate.

44

REFERENCES

Aviation Safety Reporting System. (1995). Special Request No. 3801: Checklist Use Reports [Database search]. ASRS office, Mountain View, California: Battelle.

Aviation Week and Space Technology, (1990, April 2). FAA revokes pilot licenses of captain involved in US Air 737 crash at LaGuardia. p. 53.

Davies, D. R., and Tune, G. S. (1969). Human Vigilance Performance. New York: American Elsevier Publishing Co., Inc.

Davies, D. R., and Parasuraman, R. (1982). The Psychology of Vigilance. Academic Press: London.

Degani, A. and Weiner, E. L. (1990, May). Human Factors of Flight Deck Checklists: The Normal Checklist. NASA Contract NCC2-377.

Degani, A. and Weiner, E. L. (1993). Cockpit Checklists: Concept, Design, and Use In Human Factors, 35(2), 345-354.

Dittmar, Zeileniewski, Dember, and Warm. (1986). Effects of signal regularity upon performance in simultaneous and successive vigilance tasks. In the Proceedings of the Human Factors Society 30th Annual Meeting.

Federal Aviation Administration. (1995, January). Human Performance Considerations in the Use and Design of Aircraft Checklists. Human Factors Analysis Division, Office of Integrated Safely Analysis, Associate Administrator for Aviation Safety.

Federal A viation Administration/A irman 's Information Manual. (1994). Jeppesen/Sanderson, Inc. Englewood, CO

Fisk, A. D. (1985). Automatic and controlled processing approach to interpreting vigilance performance. In the Proceedings of the Human Factors Society 29th Annual Meeting.

Fisk, A. D. and Schneider, W. (1981). Control and automatic processing during tasks requiring sustained attention: A new approach to vigilance. In the Proceedings of the Human Factors Society 25th Annual Meeting.

Hawkins, F.H. (1988). Checklists. InH. W. Orlady (Ed.) Human Factors in Flight, 2nd Edition. Ashgate Publishing.

Linde, C. and Goguen, J. (1987). Checklist Interruption and Resumption: A Linguistic Study NASA Contract Report 177460. NASA Contractor No. 11052.

45

McElhatton, J., and Drew, C. R. (1993, May/ Time Pressure as a Causal Factor in Aviation Safety Incidents: The Hurry-Up Syndrome . ASRS office, Mountain View, California: Battelle. Also in the Proceedings oj The 7th International Symposium on Aviation Psychology. Ohio State University, 1993.

Mangold, S. J., and Eldredge, D. (1993, April). An Approach to Modeling Pilot Memory and Developing a Taxonomy of Memory Errors. ASRS office, Mountain View, California: Battelle. Also in the Proceedings of The 7th International Symposium on Aviation Psychology. Ohio State University, 1993.

Mosier, K., Palmer, E., and Degani, A. (1992). Electronic checklists: Implications for decision making. In the Proceedings of the Human Factors Society 36th Annual Meeting.

Nagano, H. (1975). Human error in aviation operations. In E. L. Weiner, and D. C. Nagel (Eds.) Human Factors In Aviation (pp. 263-303). San Diego: Academic Press.

Olcott, J. W. (1991, October). Check complacency. Business and Commercial Aviation. p.134.

Miller, G. A. (1956). The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, pp. 81-97.

National Transportation Safety Board. (1969). Aviation accident recommendation A-69-012. Pan American World Airways. Boeing 7007 N99PA. Anchorage Alaska. December 12, 1968.

National Transportation Safety Board. (1988a). Aircraft accident report, NTSB/AAR-88/05, DC-9-82 N312RC, Detroit Metropolitan Airport, Romulus, Michigan. Washington, DC: NETS.

National Transportation Safety Board. (1988b). Aircraft accident report, NETS/AAA-88/06, Be-3101 N33 ICY. New Orleans International Airport, Kenner, Louisiana. Washington, DC: NTSB.

National Transportation Safety Board. (1989). Aircraft accident report, NTSB/AAR-89/04, Boeing 727-232, N473DA. Dallas-Fort Worth International Airport, Texas. Washington, DC: NTSB.

Palmer, E. and Degani, A. (1991). Electronic checklists: Evaluation of two levels of automatioa In the Proceedings of the Sixth International Symposium on Aviation Psychology, Ohio State University.

46

Sumwalt, R. (1991, March). Checking the checklist Professional Pilot. Also published in Safety On-Line Vol. 1,1.(1994). Checking the checklist. December.

USAir, (1991). Normal Operating Procedures, Boeing 737-200 Pilot's Operating Handbook. 10/111/91 pp. 3-3-1.

Wickens, C. D., and Flach, J. M. (1988). Information processing. In E. L. Weiner, and D. C. Nagel (Eds.) Human Factors in Aviation (pp. 263-303). San Diego: Academic Press.

APPENDIX A

STANDARD ATC-810 CHECKLIST VARIABLE CHECKLISTS

48

ATC 810 CHECKLIST Standard Checklist PREFLIGHT Charts, pens, notes, etc Seat STARTING Parking Break Kev Switch Master Switches Boost Pumps Alternator Circuit Breakers Throttles Landing Gear Magnetos Flaps Circuit Breakers Trims Boost Pumps Mixtures Props Cowl Flaps Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CTJMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

IEVEL OFF/CRUISE Cowl Flaps Trim Cruise Power No Smoking/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE ON OPEN 1/4 " DOWN ON (BOTH) UP CHECK SET (3 AXIS Takeoff Position1) OFF FULL RICH FULL FORWARD OPEN ROCKER SWITCH ENGAGE 1000 RPM CHECK (Yellow arc) ON (Set departure frequencies) ON (Standbv code as assigned) SET(Runwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in. Hg & 2400 RPM) OFF (Check Pressure)

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in Hg . 2100 RPM. 130 PPH Fuel OFF (if desired)

49

ATC 810 CHECKLIST Variable Checklist 1 PREFLTGHT Charts, pens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Flaps Mixtures Throttles Landing Gear Magnetos Boost Pumps Trims Circuit Breakers Cowl Flaps Alternator Circuit Breakers Props Boost Pumps Start Engines Eneine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXT CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CLIMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRUISE Cowl Flaps Trim Cruise Power No Smoking/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE ON OPEN 1/4 " DOWN ON (BOTH) UP SET (3 AXIS Takeoff Position) CHECK OFF FULL RICH FULL FORWARD OPEN ROCKER SWITCH ENGAGE 1000 RPM CHECK (Yellow arc) ON (Set departure frequencies) ON (Standbv code as assigned) SET(Runwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in. He. & 2400 RPM) OFF (Check Pressure)

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. He.. 2100 RPM. 130 PPH Fuel OFF (if desired)

50

ATC 810 CHECKLIST Variable Checklist 2 PREFLIGHT Charts, pens, notes, etc Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumrjs Cowl Flaps Props Landing Gear Throttles Circuit Breakers Flaps Magnetos Trims Alternator Circuit Breakers Boost Pumps Mixtures Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CTJMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

IJCVEL OFF/CRTTTSE Cowl Flaps Trim Cruise Power No Smokine/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE OPEN FULL FORWARD DOWN OPEN 1/4 " CHECK UP ON (BOTH) SET (3 AXIS Takeoff Position) ON OFF FULL RICH ROCKER SWITCH ENGAGE 1000 RPM CHECK (Tellow arc) ON (Set departure frequencies') ON (Standbv code as assigned) SETtRunwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He & 2575 RPM 85kts UP ("positive rate) 120 KIAS HO degree pitch attitude) SET (40 in Hg & 2400 RPM) OFF (Check Pressure)

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. Hg . 2100 RPM. 130 PPH Fuel OFF (if desired)

51

ATC 810 CHECKLIST Variable Checklist 3 PREFLIGHT Charts, pens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumps Cowl Flaps Throttles Landing Gear Magnetos Flaps Circuit Breakers Alternator Circuit Breakers Trims Boost Pumps Mixtures Props Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CLIMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRTJISE Cowl Flaps Trim Cruise Power No Smokine/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE OPEN OPEN 1/4 " DOWN ON (BOTH) UP CHECK ON SET (3 AXIS Takeoff Position) OFF FULL RICH FULL FORWARD ROCKER SWITCH ENGAGE 1000 RPM CHECK (Tellow arc) ON CSet departure frequencies') ON (Standbv code as assigned) SETfRunwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude') SET (40 in. Hg. & 2400 RPM) OFF (Check Pressure")

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. Hg.. 2100 RPM. 130 PPH Fuel OFF (if desired")

52

ATC 810 CHECKLIST Variable Checklist 4 PREFLTGHT Charts, pens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumps Throttles Flaps Cowl Flaps Props Landing Gear Magnetos Trims Alternator Circuit Breakers Circuit Breakers Boost Pumps Mixtures Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXT CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CTJMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

I.EVEI, OFF/CRUISE Cowl Flaps Trim Cruise Power No Smoking/Seat Sien

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE OPEN 1/4 " UP OPEN FULL FORWARD DOWN ON (BOTH) SET (3 AXIS Takeoff Position) ON CHECK OFF FULL RICH ROCKER SWITCH ENGAGE 1000 RPM CHECK fYellow arc} ON (Set departure frequencies) ON CStandbv code as assigned) SETfRunwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in. He. & 2400 RPM) OFF (Check Pressure)

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. He.. 2100 RPM. 130 PPH Fuel OFF (if desired)

53

ATC 810 CHECKLIST Variable Checklist 5 PREFLIGHT Charts, pens, notes, etc Seat

STARTING Parking Break Kev Switch Master Switches Circuit Breakers Boost Pumps Magnetos Trims Landing Gear Throttles Flaps Alternator Circuit Breakers Boost Pumps Mixtures Cowl Flaps Props Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CLIMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRUISE Cowl Flaps Trim Cruise Power No Smokine/Seat Sign

CHECK ADJUSTED

SET ON ON CHECK ON-CHECK PRESSURE ON (BOTH) SET (3 AXIS Takeoff Position) DOWN OPEN 1/4 " UP ON OFF FULL RICH OPEN FULL FORWARD ROCKER SWITCH ENGAGE 1000 RPM CHECK (Yellow arc) ON (Set departure frequencies) ON (Standbv code as assigned) SET(Runwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in He & 2400 RPM) OFF (Check Pressure)

fa). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in Hg. 2100 RPM. 130 PPH Fuel OFF (if desired)

54

ATC 810 CHECKLIST Variable Checklist 6 PREFLIGHT Charts, pens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumps Throttles Flaps Magnetos Trims Landing Gear Alternator Circuit Breakers Boost Pumps Mixtures Cowl Flaps Circuit Breakers Props Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CMMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRTIISE Cowl Flaps Trim Cruise Power No Smoking/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE OPEN 1/4 " UP ONrBOTH) SET (3 AXIS Takeoff Position') DOWN ON OFF FULL RICH OPEN CHECK FULL FORWARD ROCKER SWITCH ENGAGE 1000 RPM CHECK fYellow arc) ON (Set departure frequencies) ON CStandbv code as assigned1) SETfRunwav Heading") INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate") 120 KIAS HO degree pitch attitude") SET T40 in. Hs. & 2400 RPM) OFF (Check Pressure")

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. Hg.. 2100 RPM. 130 PPH Fuel OFF ("if desired")

55

ATC 810 CHECKLIST Variable Checklist 7 PREFLTGHT Charts. Dens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumps Boost Pumps Circuit Breakers Mixtures Cowl Flaps Props Trims Alternator Circuit Breakers Landing Gear Magnetos Flaps Throttles Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CLIMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRTITSE Cowl Flaps Trim Cruise Power No Smokine/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE OFF CHECK FULL RICH OPEN FULL FORWARD SET (3 AXIS Takeoff Position) ON DOWN ON (BOTH) UP OPEN 1/4 " ROCKER SWITCH ENGAGE 1000 RPM CHECK (Yellow arc) ON (Set departure frequencies') ON (Standbv code as assigned) SET(Runwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in. He. & 2400 RPM) OFF (Check Pressure)

(a). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. Hs.. 2100 RPM. 130 PPH Fuel OFF (if desired)

56

ATC 810 CHECKLIST Variable Checklist 8 PREFLTGHT Charts, pens, notes, etc. Seat

STARTING Parking Break Kev Switch Master Switches Boost Pumps Magnetos Trims Landing Gear Throttles Flaps Alternator Circuit Breakers Boost Pumps Circuit Breakers Mixtures Cowl Flaps Props Start Engines Engine Idle Oil Pressure Radios Transponder HSI Course Selector Fuel Fuel Ouantitv

TAXI CHECK Park Break Taxi No Smoking/Seatbelts Boost Pumps Transponder

TAKE-OFF/CLIMB Park Break Throttles Oil Pressure/Fuel Flow Static Check Rotate Landing Gear Accelerate Cruise/Climb Power Boost Pumps

LEVEL OFF/CRUISE Cowl Flaps Trim Cruise Power No Smoking/Seat Sign

CHECK ADJUSTED

SET ON ON ON-CHECK PRESSURE ON (BOTH) SET (3 AXIS Takeoff Position) DOWN OPEN 1/4 " UP ON OFF CHECK FULL RICH OPEN FULL FORWARD ROCKER SWITCH ENGAGE 1000 RPM CHECK (Yellow arc1) ON (Set departure frequencies) ON (Standbv code as assigned) SET(Runwav Heading) INBOARD TANKS for take-off AS PER FLIGHT PLAN

RELEASE TAKE-OFF RUNWAY HEADING ON ON ON

RELEASE FULL FORWARD PROPER RANGE 43 in He. & 2575 RPM 85kts UP (positive rate) 120 KIAS (10 degree pitch attitude) SET (40 in. He. & 2400 RPM) OFF (Check Pressure)

r5). 500' PRIOR TO ALTITUDE CLOSED NOSE DOWN/LEVEL SET 37 in. Hg.. 2100 RPM. 130 PPH Fuel OFF (if desired)

APPENDIX B

INFORMED CONSENT FORM

57

58

Informed Consent Form I, agree to participate in a research experiment on Accuracy In Fight simulator study, which is being conducted by Veronica T. Cote. I understand that participation is voluntary. I may withdraw my participation at any time and have the results of my participation returned to me, removed from the experimental records or destroyed. The following points have been explained to me: • The purpose of this experiment, which is to determine the

accuracy of pilots operating a light twin engine simulator. The benefits that I may expect to obtain from my participation are two hours of dual instruction in the ATC-810 fight simulator which will be free of any fee for the instruction or the flight.

• Participation will involve neither risk, discomfort, or stress during the study.

• The results of the study will be confidential and will not be released in any individually identifiable form without my prior consent unless required by law.

• The researcher will answer any further questions about the study, upon request.

Signature of Researcher Signature of Participant

Date Date

PLEASE SIGN BOTH COPIES. KEEP ONE AND RETURN THE OTHER TO THE RESEARCHER.

APPENDIX C

TRIP SHEETS

59

60

Please follow the directions for the following flights. Be sure to record takeoff time

(local) and the time upon reaching the checkpoint. Try to be as accurate as possible.

FLIGHT #1

Depart runway 5L at PVD. Fly runway heading until reaching 2000'. Turn right, proceed

direct to the FLR (Fall River NDB). Report reaching FLR NDB and record time.

Com#l 120.7 Nav#l 109.3 Nav#2 115.6 ADF 406 Squawk 5637

Take-Off time: Checkpoint time:

FLIGHT #2

Depart runway 5L at PVD. Fly runway heading until reaching 2000'. Turn left, intercept

PVD 360° radial until reaching 15 DME. Report reaching the 15 DME and record time.

Com#l 120.7 Nav#l 109.3 Nav#2 115.6 ADF 406 Squawk 5637

Take-Off time: Checkpoint time:

FLIGHT #3

Depart runway 5 at EWB. Fly runway heading until reaching 1400'. Turn left, proceed

direct to FLR (Fall River NDB). Report reaching and record time.

Com#l 118.10 Nav#l 109.7 Nav#2 115.6 ADF 406 Squawk 5637

Take-Off time: Checkpoint time:

FLIGHT #4

Depart runway 5 at EWB. Fly runway heading until reaching 1800'. Proceed direct to

TAN (Taunton NDB). Report reaching TAN NDB and record time.

Com #1 118.10 Nav#l 109.7 Nav#2 115.6 ADF 227 Squawk 5637

Take-Off time: Checkpoint time:

61

FLIGHT #5

Depart runway 5 at EWB. Fly runway heading until reaching 1000' Turn right and

proceed direct to the MVY VOR. Report reaching the MVY VOR and record time.

Com#l 118.10 Nav#l 109.7 Nav#2 114.5 ADF 274 Squawk 5637

Take-Off time: Checkpoint time:

FLIGHT #6

Depart runway 6 at ACK. Fly runway heading until reaching 2500', then turn right and

proceed to the ACK VOR. Continue to climb to 4000. Report reaching the ACK VOR

and record time.

Com#l 118.30 Nav#l 109.1 Nav#2 116.2 ADF 248 Squawk 5637

Take-Off time: Checkpoint time:

FLIGHT #7

Depart runway 6 at ACK. Fly runway heading until reaching 1000'. Turn left, proceed

direct to the MVY VOR at 1000'. Report reaching the MVY VOR and record time.

Take-Off time: Checkpoint time:

FLIGHT #8

Depart runway 24 at MVY. Fly runway heading until reaching 800'. Turn left, proceed

direct to the ACK VOR while continuing to climb to 2800' Report reaching ACK VOR

and record time.

Take-Off time: Checkpoint time:

Thank you for your assistance in this research effort. It has been greatly appreciated.

APPENDIX D

LIST OF RANDOM FAULTS

62

63

Faults set during trials 7 and 8

The following are the faults that were randomly selected and set during trials 7 and 8 for both the experimental and the control groups.

Faults 1 2 3 4 5 6 7 8

Gear Fuel pumps Roll Trim Cowl Flaps Oil Pressure Fuel Selector Circuit Breakers Flaps

Set 11 10 13 8 6 12 11 9

Missed 2 1 3. 0 1 1 4 0

APPENDIX E

RAW DATA

64

65

Raw data for control and experimental groups Time is given is seconds. Faults are numbered, 1-8, and correspond with faults in Appendix D.

Control Group

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 200

7,6

1,3 Trial 2 195

Trial 3 170

C I

4-14

170

4

Errors found 7

Errors found 8 Trial 4 170

Trial 5 130

6

1 Trial 6 110

Trial 7 102

Trial 8 100

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8 Time Trial 1

145

6,3

1,8 Trial 2 122

Trial 3 160

C2 4-18

250

26

Errors found 7

Errors found 8 Trial 4 127

Trial 5 125

6,3

1,8 Trial 6 150

Trial 7 150

Trial 8 145

Subject

Date

Total Flight Hours HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 132

3,4

1,2 Trial 2 135

Trial 3 118

C3

4-19

130

20

Errors found 7

Errors found 8 Trial 4 110

Trial 5 107

3,4

1,2 Trial 6 106

Trial 7 113

Trial 8 147

Control Group

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 145

3,1

6,8

Trial 2 118

Trial 3 127

C 4

4-19

225

10

Errors found 7

Errors found 8

Trial 4 103

Trial 5 80

3,1

6,8

Trial 6 85

Trial 7 160

Trial 8 110

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 115

8,3

2,5

Trial 2 110

Trial 3 98

C 5

4-20

215

12

Errors found 7

Errors found 8

Trial 4 85

Trial 5 80

8

5

Trial 6 90

Trial 7 75

Trial 8 100

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 182

2,6

7,1 Trial 2 145

Trial 3 120

C 6

4-24

130

20

Errors found 7

Errors found 8

Trial 4 110

Trial 5 107

2,6

7,1 Trial 6 106

Trial 7 140

Trial 8 147

67

Control Group

Subject Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 178

3,6

7,2 Trial 2 98

Trial 3 107

C7

4-26

120

3

Errors found 7

Errors found 8 Trial 4 100

Trial 5 97

3,6

7,2 Trial 6 105

Trial 7 124

Trial 8 120

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 160

5,8

4,7 Trial 2 85

Trial 3 102

C8

4-27 220

54

Errors found 7

Errors found 8 Trial 4 95

Trial 5 70

5,8 4,7

Trial 6 70

Trial 7 180

Trial 8 102

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 200

5,1 3,6

Trial 2 220

Trial 3 195

C9

5-1

800

3

Errors found 7

Errors found 8 Trial 4 180

Trial 5 195

1

3,6 Trial 6 210

. . <

Trial 7 185

Trial 8 210

68

Control Group

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 200

1,2 7,5

Trial 2 130

Trial 3 110

CIO

5-2

250

3

Errors found 7

Errors found 8 Trial 4 115

Trial 5 105

2

7,5 Trial 6 100

Trial 7 95

Trial 8 116

Experimental Group

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 90

8,3

4,1 Trial 2 95

Trial 3 80

E l

4-10

960

7

Errors found 7

Errors found 8 Trial 4 85

Trial 5 80

8,3

4,1 Trial 6 85

Trial 7 104

Trial 8 100

Subject

Date

Total Flight Hours HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 135

8,3

4,1 Trial 2 125

Trial 3 127

E2

4-11

110

20

Errors found 7

Errors found 8 Trial 4 100

Trial 5 120

8,3

4,1 Trial 6 90

Trial 7 127

Trial 8 150

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 128

7,5

2,6 Trial 2 122

Trial 3 100

E3

4-11

330

18

Errors found 7

Errors found 8 Trial 4 75

Trial 5 125

7,5

2 Trial 6 110

Trial 7 150

Trial 8 145

Experimental Group

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 67

2,4

3,8

Trial 2 75

Trial 3 82

E 4

4-13

300

29

Errors found 7

Errors found 8

Trial 4 85

Trial 5 90

2,4

3,82

Trial 6 75

Trial 7 140

Trial 8 110

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 90

1,3

7,8

Trial 2 96

Trial 3 82

E 5

4-13

215

40

Errors found 7

Errors found 8

Trial 4 88

Trial 5 70

1

8

Trial 6 75

Trial 7 80

Trial 8 96

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 82

6,4

1,7

Trial 2 83

Trial 3 88

E 6

4-18

195

4

Errors found 7

Errors found 8

Trial 4 90

Trial 5 85

6,4

0

Trial 6 91

Trial 7 159

Trial 8 137

71

Experimental Group

Subject

Date

Total Flight Hours

Hours inATC-810

Errors set 7

Errors set 8

Time Trial 1 162

4,8

3,5

Trial 2 126

Trial 3 112

E 7

4-20

40

3

Errors found 7

Errors found 8

Trial 4 135

Trial 5 87

4,8

3,5

Trial 6 90

Trial 7 98

Trial 8 114

Subject

Date

Total Flight Hours

Hour s inATC-810

Errors set 7

Errors set 8

Time Trial 1 160

8,4

7,2

Trial 2 147

Trial 3 140

E 8

4-25

55

10

Errors found 7

Errors found 8

Trial 4 220

Trial 5 120

8,4

7,2

Trial 6 123

Trial 7 122

Trial 8 125

Subject

Date

Total Flight Hours

Hour s inATC-810

Errors set 7

Errors set 8

Time Trial 1 156

1,2

7,6

Trial 2 115

Trial 3 100

E 9

4-26

270

6

Errors found 7

Errors found 8

Trial 4 100

Trial 5 95

1,2

7,6

Trial 6 80

Trial 7 116

Trial 8 100

72

Experimental Group

Subject

Date

Total Flight Hours

HoursinATC-810

Errors set 7

Errors set 8

Time Trial 1 90

6,4 5,7

Trial 2 95

Trial 3 95

E10

5-2

320

26

Errors found 7 Errors found 8

Trial 4 95

Trial 5 92

6,4 5

Trial 6 93

Trial 7 100

Trial 8 98

APPENDIX F

FLIGHT HOURS, SIMULATOR HOURS OF SUBJECTS CHECKLIST COMPLETION TIMES AND FAULTS MISSED

73

Total flight and simulator hours of the subjects

Control Group

Subject

1 2 3 4 5 6 7 8 9 10

X mean

Flight Hours

170 250 130 225 215 130 120 220 800 250 251

ATC-810 Hours

4 26 20 10 12 20 3 54 3 3

14.4

Experimental Group

Subject

1 2 3 4 5 6 7 8 9 10

X mean

Flight Hours

960 110 330 300 215 195 40 55

270 320

279.5

ATC-810 Hours

7 20 18 29 40 4 3 10 6 26

16.3

75

Checklist Completion Times and Faults Missed All times are in seconds.

Control Group

Subject

1 2 3 4 5 6 7 8 9 10

Mean

Trial 1

200 145 132 145 115 182 178 160 200 200

165.7

Trial 2

195 122 135 118 110 145 98 85

220 130

135.8

Trial 3

170 160 118 127 98 120 107 102 195 110

130.7

Trial 4

170 127 110 103 85 110 100 95 180 115

119.5

Trial 5

130 125 107 80 80 107 97 70 195 105

109.6

Trial 6

110 150 106 85 90 106 105 70

210 100

113.2

Trial 7

102 150 113 160 75 140 124 180 185 95

132.4

Trial 8

100 145 147 110 100 147 120 102 210 116

129.7

Faults found Trial

7 1 2 2 2 1 2 2 2 1 1 4

Faults found Trial

8 1 2 2 2 1 2 2 2 2 2 2

Experimental Group

Subject

1 2 3 4 5 6 7 8 9 10

Mean

Trial 1

90 135 128 67 90 82 162 160 156 90 116

Trial 2

95 125 122 75 96 83 126 147 115 95 107

Trial 3

80 127 100 82 82 88 112 140 100 95

100.6

Trial 4

85 100 75 85 88 90 135 220 100 95

107.3

Trial 5

80 120 125 90 70 85 87 120 95 92

96.4

Trial 6

85 90 110 75 75 91 90 123 80 93

91.2

Trial 7

104 127 150 140 80 159 98 122 116 100

119.6

Trial 8

100 150 145 110 96 137 114 125 100 98

117.5

Faults found Trial

7 2 2 2 2 1 2 2 2 2 2

Faults found Trial

8 2 2 1 2 1 0 2 2 2 1

1 1 5 |