emerging results from the “safer” project

Emerging Results From the Emerging Results From the “SAFER” Project“SAFER” Project

Dr. Wayne A. Dornan Dr. Wayne A. Dornan Middle Tennessee State Middle Tennessee State

UniversityUniversityMurfreesboro, TN Murfreesboro, TN

CollaboratorsCollaborators

Dr. Paul Craig (Principal Dr. Paul Craig (Principal Investigator)Investigator)

Dr. John Bertrand (Co-Principal Dr. John Bertrand (Co-Principal Investigator) Investigator) Steve GossettSteve GossettKim ThorsbyKim Thorsby

PILOT ERROR PILOT ERROR CONTINUES TO PLAGUE CONTINUES TO PLAGUE

THE INDUSTRYTHE INDUSTRY

Contributing FactorsContributing Factors

Pilots Missing a Problem Pilots Missing a Problem

Recognizing a Problem to LateRecognizing a Problem to Late

Misidentifying a Problem Misidentifying a Problem

Not Recognizing that there is an Not Recognizing that there is an Immediate Need to Deal with the ProblemImmediate Need to Deal with the Problem

Not Utilizing Their ResourcesNot Utilizing Their Resources

Another Contributing Another Contributing FactorFactor

In aviation, the importance of critical In aviation, the importance of critical decision making skills decision making skills can notcan not be over- be over-emphasizedemphasizedIndeed, the safety of each flight depends Indeed, the safety of each flight depends on it !on it !

Factors Influencing Critical Factors Influencing Critical Decision Making SkillsDecision Making Skills

One possible contributing factor something One possible contributing factor something psychologists have known for years.psychologists have known for years.

Humans consistently are found not to give enough Humans consistently are found not to give enough importance to new evidenceimportance to new evidence

Can you think of an example? Can you think of an example? Inadvertent IMC (176 seconds to live)Inadvertent IMC (176 seconds to live)

Effective Pilot Training Effective Pilot Training Programs Make Pilots Aware of Programs Make Pilots Aware of

this Phenomenonthis Phenomenon

The Decide Model The Decide Model

Was developed to help pilots organize Was developed to help pilots organize their thoughts to prevent overlooking their thoughts to prevent overlooking factors that may be important factors that may be important

Recent Issues That are Emerging Recent Issues That are Emerging

““Out-of-the-Loop” phenomenon Out-of-the-Loop” phenomenon

ComplacencyComplacency

Lack of Knowledge of Automation Lack of Knowledge of Automation

Poor Situational AwarenessPoor Situational Awareness

Poor Situational AssessmentPoor Situational Assessment

HOW DO THE AIRLINES HOW DO THE AIRLINES DEAL WITH THIS ?DEAL WITH THIS ?

MEMORY ITEMS MEMORY ITEMS

CHECKLISTSCHECKLISTS

SCENARIO BASED TRAININGSCENARIO BASED TRAINING

““FITS”FITS”FAA Industry Training StandardsFAA Industry Training Standards

The initiative has been guided by a visionary group The initiative has been guided by a visionary group of FAA administrators, educators, and industry of FAA administrators, educators, and industry

““SAFER”SAFER”

SSATS ATS AAerospaceerospaceFFlight light EEducation ducation RResearchesearch

NASA Research Cooperative AgreementNASA Research Cooperative AgreementNCCI - 03032NCCI - 03032

“FAA Industry Training StandardsScenario Based

Private / Instrument Syllabus forTechnically Advanced Piston Aircraft”

was approved by FAA Headquartersin Washington D.C. and at the Nashville FAA Flight Standards

District Office, as a part of MTSU’s existingAir Agency Certificate

FAA Exemption 8456 to 61.65(a)(1)

Sixteen MTSU Sixteen MTSU students began their students began their flight training in flight training in “glass” in DA-40 “glass” in DA-40 aircraft using the aircraft using the FAA-approved FAA-approved training program in training program in August 2004August 2004

Private Pilot BottlenecksPrivate Pilot BottlenecksArchival data 1999-2004Archival data 1999-2004

00.5

11.5

22.5

33.5

44.5

Target TimeAverage Time

Instrument Pilot BottlenecksInstrument Pilot BottlenecksArchival data 1999-2004Archival data 1999-2004

0

1

2

3

4

5

6

7

IFR1 IFR4 IFR7 IFR10 IFR13 IFR16 IFR19 IFR22 IFR25 IFR28

Target Time

Average Time

Setbacks ComparisonSetbacks ComparisonCaution: Preliminary data using small numbersCaution: Preliminary data using small numbers

0

24

68

1012

14

Start Presolo Pvt X-C IFR

SAFER

Traditional

Traditional SAFER Pre Solo 77 of 449 17.1% 59 of 97 60.8% Pvt & X-C 169 of 449 37.6% 15 of 97 15.4% Instrument 203 of 449 45.2% 23 of 97 23.7%

The results are still The results are still preliminary! preliminary!

However, early data tends to However, early data tends to indicate that pilots have fewer indicate that pilots have fewer

setbacks over the entire VFR/IFR setbacks over the entire VFR/IFR training using new technology training using new technology

and a scenario-based syllabusand a scenario-based syllabus

The first student in our second cohort The first student in our second cohort passed on her combined check ride on passed on her combined check ride on

June 15June 15thth , 2005 , 2005

She had a total of 55 airplane hours !She had a total of 55 airplane hours !

IF WE ARE GOING TO BE IF WE ARE GOING TO BE CERTIFICATING PILOTS WITH CERTIFICATING PILOTS WITH

FEWER HOURSFEWER HOURS

WHAT KIND OF PILOTS WILL THEY WHAT KIND OF PILOTS WILL THEY

BE ?BE ?

PHASE TWOPHASE TWO“ “ THE EFFECTS OF A “FITS” THE EFFECTS OF A “FITS” TRAINING PROGRAM THAT TRAINING PROGRAM THAT

EMPHASIZES SCENARIO BASED EMPHASIZES SCENARIO BASED FLYING ON PILOT DECISION FLYING ON PILOT DECISION

MAKING SKILLS “ MAKING SKILLS “

METHODOLOGYMETHODOLOGY

16 Students enrolled in the “SAFER” 16 Students enrolled in the “SAFER” project: COMBINED GROUPproject: COMBINED GROUP24 Pilots who had obtained their 24 Pilots who had obtained their instrument rating within the last 3 months instrument rating within the last 3 months TRADITIONAL GROUPTRADITIONAL GROUP

METHOLODOLOGY METHOLODOLOGY

The “combined” group were administered The “combined” group were administered pencil and paper tests throughout their pencil and paper tests throughout their flight training flight training

The “traditional group were administered The “traditional group were administered pencil and paper tests within three months pencil and paper tests within three months following the successful completion of their following the successful completion of their check ride check ride

METHODOLOGYMETHODOLOGYAll “SAFER” students also enrolled in a ground class that All “SAFER” students also enrolled in a ground class that combined both private pilot and instrument knowledgecombined both private pilot and instrument knowledgeThis class placed a major emphasis on This class placed a major emphasis on Automation (e.g. mode awareness, automation traps)Automation (e.g. mode awareness, automation traps)Situational AwarenessSituational AwarenessGPS technologyGPS technologyGPS programming skillsGPS programming skillsCBT using real flight scenario’s CBT using real flight scenario’s Critical thinking skills using NTSB reports of fatal aircraft Critical thinking skills using NTSB reports of fatal aircraft accidents as examplesaccidents as examples

DEMOGRAPHICSDEMOGRAPHICS

AGE: AGE: CombinedCombined 19.6 19.6 ++ 1.6 1.6

TraditionalTraditional 20.7 + 1.720.7 + 1.7

DEMOGRAPHICSDEMOGRAPHICS

Total Airplane Hours:Total Airplane Hours:CombinedCombined 90.8 90.8 ++ 5.0 *** 5.0 ***TraditionalTraditional 176.8 176.8 ++ 12 12

Total Flight Hours: Total Flight Hours: CombinedCombined 90.8 90.8 ++ 5.0 *** 5.0 ***TraditionalTraditional 188.75 188.75 ++ 10 10

*** = p <0.01*** = p <0.01

DEMOGRAPHICSDEMOGRAPHICS

Total Instrument Hours:Total Instrument Hours:CombinedCombined 44.3 44.3 ++ 2.0 2.0TraditionalTraditional 38.3 38.3 ++ 3.0 3.0

Total Actual Hours: Total Actual Hours: CombinedCombined 6.1 6.1 ++ 1.8 1.8TraditionalTraditional 4.6 4.6 ++ 0.7 0.7

DEMOGRAPHICSDEMOGRAPHICS

Total PIC Hours:Total PIC Hours:

CombinedCombined 6.1 6.1 ++ 0.6 *** 0.6 ***

TraditionalTraditional 125.6 125.6 ++ 14.0 14.0

*** = p < 0.01*** = p < 0.01

Personal IFR Comfort Personal IFR Comfort Questionnaire Questionnaire

How comfortable are you to fly alone in How comfortable are you to fly alone in the IFR environment? the IFR environment?

How comfortable are you to fly alone in How comfortable are you to fly alone in IMC ? IMC ?

How comfortable are you to shoot an ILS How comfortable are you to shoot an ILS approach to minimums ? approach to minimums ?

What are your “personal minimums” ? What are your “personal minimums” ?

ResponsesResponses

1 = Not comfortable 1 = Not comfortable

2 = Somewhat comfortable2 = Somewhat comfortable

3 = comfortable3 = comfortable

4 = very comfortable4 = very comfortable

5 = Absolutely comfortable, no problem!5 = Absolutely comfortable, no problem!

RESULTSRESULTS

Not Comfortable flying alone in the IFR Not Comfortable flying alone in the IFR environmentenvironment

Traditional 18 %Traditional 18 %

Combined 0 % ****Combined 0 % ****

*** = p < 0.01*** = p < 0.01

RESULTSRESULTS

Not Comfortable flying alone in IMCNot Comfortable flying alone in IMC

Traditional 38 %Traditional 38 %

Combined 14 % ****Combined 14 % ****

*** = p < 0.01*** = p < 0.01

RESULTSRESULTS

Not Comfortable shooting an ILS to Not Comfortable shooting an ILS to minimumsminimums

Traditional 16 %Traditional 16 %

Combined 0 % ****Combined 0 % ****

*** = p < 0.01*** = p < 0.01

RESULTSRESULTS

Would you feel comfortable using a Would you feel comfortable using a GPS when flying IMCGPS when flying IMC

Traditional = Yes (48 %) ***Traditional = Yes (48 %) ***

Combined = Yes (100 %) Combined = Yes (100 %)

**** = p < 0.01 **** = p < 0.01

RESULTSRESULTS

What are your personal minimums ? What are your personal minimums ?

Visibility: Visibility:

Traditional 2.07 + .18Traditional 2.07 + .18

Combined 3.50 + .06 ****Combined 3.50 + .06 ****

*** = p < 0.01*** = p < 0.01

RESULTSRESULTS

What are your personal minimums ? What are your personal minimums ?

Clouds: Clouds:

Traditional 1,350 + 223Traditional 1,350 + 223

Combined 2,100 + 585 ****Combined 2,100 + 585 ****

*** = p < 0.01*** = p < 0.01

RESULTSRESULTS

What are your personal minimums ? What are your personal minimums ?

Visibility: Never Thought about it Visibility: Never Thought about it !!

Traditional 68 %Traditional 68 %

Combined 18 % ****Combined 18 % ****

*** = p < 0.01*** = p < 0.01

Self-Efficacy: TAA Self-Efficacy: TAA

I have confidence in my ability to fly a TAAI have confidence in my ability to fly a TAA1 = to no extent1 = to no extent 5 = to a great extent 5 = to a great extent

Students were assessed at: Beginning of Students were assessed at: Beginning of flight training, Mid-flight training, and at flight training, Mid-flight training, and at their completiontheir completion

RESULTSRESULTS

Beginning 4 – 5 = 58 % *** (3.70)Beginning 4 – 5 = 58 % *** (3.70)Mid-training 4 – 5 = 100 % (4.75)Mid-training 4 – 5 = 100 % (4.75)EndEnd 4 – 5 = 100 % (4.75)4 – 5 = 100 % (4.75)

Did not complete the training: 3.10 *Did not complete the training: 3.10 *Did complete the training: 3.70Did complete the training: 3.70

*** = p < 0.01*** = p < 0.01* = p > 0.05 * = p > 0.05

Self-Efficacy: Single Pilot Resource Self-Efficacy: Single Pilot Resource Management (SRM)Management (SRM)

I have confidence in my ability to perform SRMI have confidence in my ability to perform SRM

1 = to no extent1 = to no extent 5 = to a great extent 5 = to a great extent

Students were assessed at: Beginning of flight Students were assessed at: Beginning of flight training, Mid-flight training, and at their training, Mid-flight training, and at their completioncompletion

RESULTSRESULTS

Beginning 4 – 5 = 65 % *** (3.80)Beginning 4 – 5 = 65 % *** (3.80)

Mid-training 4 – 5 = 100 % (4.75)Mid-training 4 – 5 = 100 % (4.75)

EndEnd 4 – 5 = 80 % (4.16)4 – 5 = 80 % (4.16)

Did not complete the training: 2.70 *Did not complete the training: 2.70 *

Did complete the training: 3.80Did complete the training: 3.80

*** = p < 0.01*** = p < 0.01

* = p > 0.05 * = p > 0.05

Other Data Analysis Pending Other Data Analysis Pending

Any relationship between cognitive Any relationship between cognitive demands and “choke points” demands and “choke points”

Any relationship between self-efficacy and Any relationship between self-efficacy and quality of TAA pilotquality of TAA pilot

Follow up study at 3-6 months Follow up study at 3-6 months

SUMMARYSUMMARY

1)1) Are results indicate that pilots have fewer Are results indicate that pilots have fewer setbacks over the entire VFR/IFR training using setbacks over the entire VFR/IFR training using

the FITS syllabus the FITS syllabus 2)2) Pilots trained using a combined private / Pilots trained using a combined private /

instrument ground school that focuses on instrument ground school that focuses on decision making, automation issues, and decision making, automation issues, and

navigational issues in TAA aircraft, in navigational issues in TAA aircraft, in combination with a FITS flight training syllabus combination with a FITS flight training syllabus

emphasizing scenario based flying, are:emphasizing scenario based flying, are:

1)1) More comfortable with their automationMore comfortable with their automation2)2) More comfortable with their IFR skillsMore comfortable with their IFR skills

3)3) More conservative with IFR decision making More conservative with IFR decision making

QUESTIONSQUESTIONS