pilot training evaluation techniques - nbaa training evaluation techniques chris warton - bombardier...
TRANSCRIPT
Pilot Training Evaluation TechniquesChris Warton - Bombardier Aircraft Training - October 10th, 2012
3
Training Evaluation Prerequisites
• Appropriateness and Relevance
• Standardization:
– Contents and execution
• Validity:
– Accuracy of tasks with the relevant learning goals
4
Training Evaluation Prerequisites
• Reliability:
– Constant accuracy with comparable results
• Objectivity:
– Same results with application by different evaluators
• Transparency and Reasonableness:
– No traps and tricks
• Fairness:
– Equal treatment of candidates
5
Present Status
• Description of performance in the words of the individual instructor orevaluator
• Subjective view due to different priorities
• Interpretation of assessments
• CRM criteria not well defined
– (PTS for ATP)“Pass/Fail judgments based solely on CRM/SRM issues mustbe carefully chosen since they may be entirely subjective. Those Pass/Failjudgments, which are not subjective, apply to CRM-related procedures inFAA-approved operations manuals that must be accomplished, such asbriefings to other crewmembers. In such cases, the operator (or the aircraftmanufacturer) specifies what should be briefed and when the briefingsshould occur.“
6
Present Status
• Quality description – Only two choices
– Met standards
– Did not meet standards
• Trend not recognizable
• Flaws in the system not identifiable
7
Grading in other areas of life…
• Grade School (A-B-C-D-F)
• Collage (SAT and GPA)
• Customer Surveys (0.0 to 5.0 scale)
• Gallup polls
• AIN and Pro Pilot
8
How do we grade our crews?
Altitude
Heading
+100 ft
-100 ft
+10°+10°
9
Where they actually performed
Better performance
Worse performance
UnsatisfactoryPerformance
SatisfactoryPerformance
10
Where the same pilots were graded
UnsatisfactoryPerformance
Better performance
Worse performance
SatisfactoryPerformance
11
Behavioral Marker Systems
• NOTECHS
– The European Joint Aviation Requirements (JAR) require thetraining and assessment of pilots’ CRM skills. JAR Ops NPA 16states: “The flight crew must be assessed on their CRM skills inaccordance with a methodology acceptable to the Authority andpublished in the Operations Manual. The purpose of such anassessment is to: Provide feedback to the individual and serve toidentify retraining; and be used to improve the CRM trainingsystem”.
12
Behavioral Marker Systems
• UT at Austin Line Operations Safety Audit (LOSA)
– First developed for research and training purposes in the aviationindustry and the best-known example is the University of Texas (UT)Behavioral Markers developed by the University of Texas HumanFactors Research Project.
– The original behavioral marker system in the U.S. originated in theUniversity of Texas Human Factors Research Project (then calledthe NASA/UT Project) in the late 1980s.
13
Identified Necessary Actions
• Define detailed observable rating criteria
• By observing frequencies of behavior the progress and theabsolute performance to be described
• Make transparent for the trainee
• Differentiate quality of performance
• Define behavioral markers to describe the interpersonalcompetence
14
How do we get there?
15
UT system
16
NOTECHS
17
Standards
• FAA PTS
18
Standards - PTS
19
Standards
• FAA PTS
– Unsatisfactory Performance
20
Scoring System options
PTS
FAIL PASS
BAT TRAINING
Fail Pass
TRANSPORT CANADA
BelowStandard
BasicStandard Standard
AboveStandard
NOTECHS
VERYPOOR POOR
ACCEPTABLE GOOD
VERYGOOD
BAT PROPOSED MARKING
BelowStandard /
No Progress
BelowStandard /
NormalProgress
CompetentStandard
AboveStandard
HighStandard
Better performanceWorse performance
21
Grading Criterion
• Appropriateness and Relevance
• Standardization:
– Contents and execution
• Validity:
– Accuracy of tasks with the relevant learning goals
22
Grading Criterion
• Reliability:
– Constant accuracy with comparable results
• Objectivity:
– Same results with application by different evaluators
• Transparency and Reasonableness:
– No traps and tricks
• Fairness:
– Equal treatment of candidates
23
Aircraft Handling
Below Standard / NoProgress
Below Standard /Normal Progress
Competent Standard Above Standard High Standard
Frequent or sustaineddeviations outsideallowable tolerances.Lack of positive aircraftcontrol. Significantweakness in basictechnique. Frequentmistakes made inselection of basicmodes of automatedsystems
Deviations to the limitsof allowabletolerances, slowlycorrected, ORoccasionally exceededtolerances,immediately corrected.Evidence of poortechnique. Basicselection of automatedsystems correct, buterrors occur due tolack of understanding,or interaction withother systems.
Aircraft manipulatedwith some deviationsaway from targetparameters, quicklyrecovered. Clearevidence ofunderstanding correctmanipulativetechnique. Mostlyappropriate use ofautomated systems.
Manipulation accurate,with only occasionalrare variation fromtarget parameters,quickly corrected.Correct techniques,with good anticipationof changes. Carries outalmost all tasks usingthe automated systemscorrectly. Understandsunderlying principlesand limitations.
Manipulation soaccurate that there areno deviations fromtarget parameters.Clear mastery ofcorrect techniques atall times. Completeunderstanding andtotally appropriate useof automated systemsat all times.
Aircraft HandlingSafe, Efficient, ComfortableAutomatic/Manual FlightNon Normals/EmergenciesManages Errors
24
Current Grading – No ability to see atrend
2010 2011 2012 2013
“SAT”
“UNSAT”
RecordedPerformance
ActualPerformance
Initial Course
RecurrentCourses
25
Future Evaluation – Ability to see atrend
2010 2011 2012 2013
RecordedPerformance
ActualPerformance
Initial Course
RecurrentCourses
“SAT”
“UNSAT”
26
Which pilot grading is best for us?
Better performance
Worse performance
UnsatisfactoryPerformance
SatisfactoryPerformance
Better performance
Worse performance
UnsatisfactoryPerformance
SatisfactoryPerformance
