Monitoring Normal OperationsMonitoring Normal Operationsand the Line Operations and the Line Operations

Safety Audit (LOSA): Safety Audit (LOSA): The Perspective of ICAO The Perspective of ICAO

Captain Dan MaurinoCaptain Dan MaurinoFlight Safety and Human Factors – ICAOFlight Safety and Human Factors – ICAO

Royal Aeronautical Society Conference onRoyal Aeronautical Society Conference onMitigating Human ErrorMitigating Human Error

London, 15 October 2003London, 15 October 2003

System System designdesign

Baseline performanceBaseline performance

BaselineBaselineperformanceperformance

OperationalOperationalperformanceperformance

““PracticalPracticalDrift”Drift”

OperationalOperationaldeploymentdeployment

When the Book Goes Out of the WindowWhen the Book Goes Out of the Window

Raiding the Lost ArkRaiding the Lost Ark

NaturalNaturalPerformancePerformance

““Angel” Angel” PerformancePerformance

RegulatorRegulator NobodyNobodyLine Line

Checks Checks

LOSA LOSA

A Day at the OfficeA Day at the OfficeTo achieve daily safety and production goals, flight crews mustTo achieve daily safety and production goals, flight crews must

Threat Threat ManagementManagement

Error ManagementError Management

Undesired Aircraft Undesired Aircraft State ManagementState Management3 Manage aircraft deviations

LOSA LOSA IndicesIndices

Manage operational Manage operational complexitycomplexity

1

Manage their own errorsManage their own errors2

Not flight crew errorsNot flight crew errors Not deficiencies in the systemNot deficiencies in the system

Factors external to the flight deck that Factors external to the flight deck that increase complexity of flight operationsincrease complexity of flight operations

Must be managed by the flight crewMust be managed by the flight crew Threats = Red flagsThreats = Red flags

LOSA IndexLOSA Index: Threats: Threats

Threats Threats AreAre the Context the Context

WeatherWeather

MaintenanceMaintenance

GroundGround CrewCrew

CabinCabin CrewCrew

PassengerPassenger eventsevents ATCATC

TerrainTerrain

SimilarSimilar callcall signssigns

TimeTime pressurespressures

HeavyHeavy traffictraffic

UnfamiliarUnfamiliar airportsairports

AutomationAutomation eventsevents MissedMissed approachesapproaches

FlightFlight diversionsdiversions

SystemSystem

malfunctionsmalfunctions

DistractionsDistractions

Threat Types (T otal = 300 LOSA flights )

Threat Prevalence (Perc entage of LOSA flights w ith a threat)

1 0 % 2 0 % 3 0 % ` 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

All T hreats

Adv ers e Weather

AT C

Airline Operational Pres s ure

Airc raft M alfunc tionM EL Item s

97%

58%

57%

31%

22%

Threat Types (T otal = 300 LOSA flights )

Threat M ismanagement (Perc entage of LOSA flights w ith a m is m anaged threat)

1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

All T hreats

Adv ers e Weather

AT C

Airline Operational Pres s ure

Airc raft M alfunc tionsM EL Item s

5%

13%

13%

41%

3%

LOSA IndexLOSA Index: [Expert] Errors : [Expert] Errors

Error Types(T otal = 300 LO SA flights )

Error Prevalence (Perc entage of LO SA flights w ith an error)

1 0 % 2 0 % 3 0 % ` 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

All Errors

M anual A irc raft Handling

C hec k lis ts

C allout

81%

34%

19%

10%

Error Types(T otal = 300 LOSA flights )

Error M ismanagement (Perc entage of LOSA flights w ith a m is m anaged error)

1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

All Errors

M anual A irc raft Handling

C hec k lis ts

C allout5%

27%

9%

42%

LOSA Provides Data LOSA Provides Data OnlyOnly

ProficiencyProficiencyKnowledgeKnowledgeCRM SkillsCRM Skills

Procedural ComplianceProcedural Compliance

Crew Strengths and Crew Strengths and WeaknessesWeaknesses

Corporate CultureCorporate CultureAirports / Navigational AidsAirports / Navigational Aids

Aircraft Design and AutomationAircraft Design and AutomationStandards / Maintenance / Standards / Maintenance /

TrainingTrainingCrew SupportCrew Support

Systemic Strengths and Systemic Strengths and WeaknessesWeaknesses

LOSA Raw DataLOSA Raw Data

LOSA ReportLOSA Report

80%

13%7%

0%

20%

40%

60%

80%

Visual Non-Prec Prec

NCA: Type Of Approach

47%

27%20%

0%

10%

20%

30%

40%

50%

App Chng Rwy Chng Slam Dnk

NCAs Affected By ATC (47%)

93%

7%

0%

50%

100%

Force a Ldg Go - Around

NCA - Crew Response At The Window

47% 53%

0%

20%40%60%80%

100%

Capt FO

NCA: Pilot Flying

Targets for ImprovementTargets for ImprovementStabilized ApproachesStabilized ApproachesChecklistsChecklistsProcedural errorsProcedural errorsAutomation errorsAutomation errorsATC communicationsATC communicationsIntentional non-compliance Intentional non-compliance

SCP: Data Develops Into InformationSCP: Data Develops Into Information

Re-measureRe-measure

ProficiencyProficiencyKnowledgeKnowledgeCRM SkillsCRM Skills

Procedural ComplianceProcedural Compliance

Crew Strengths and WeaknessesCrew Strengths and Weaknesses

Corporate CultureCorporate CultureAirports / Navigational AidsAirports / Navigational Aids

Aircraft Design and AutomationAircraft Design and AutomationStandards / Maintenance / TrainingStandards / Maintenance / Training

Crew SupportCrew Support

Systemic Strengths and WeaknessesSystemic Strengths and Weaknesses

Safety Safety ChangeChangeProcessProcess(SCP)(SCP)

LOSA Raw DataLOSA Raw Data

LOSA ReportLOSA Report

LOSA: Non-Conforming Approaches Percent Improvement Over 1996 (Approaches Outside Required

1000 ft or 500 ft Window)

1996 - ALL 1998 - L.A. 1999 - PACIFIC 2000 - ALL

Reduced

49% Reduced59%

Reduced

59%

LOSA 1996 vs. 2000 Checklist ErrorsLOSA 1996 vs. 2000 Checklist Errors

11% of the flights9% of the flightsLOSA 2000

385 flights

Missed itemsWrong response to a challengeWrong checklist performedUnintentional checklist omission

Performed from memoryNonstandard phraseologySelf initiated

Not called “complete”Intentional checklist omission

Checklist Error

Examples

33% of the flights

Flights with protocol checklist errors

9% of the flights

Flight with unintentional checklist errors

LOSA 1996389 flights

LOSA 2000 indicated a significant drop inLOSA 2000 indicated a significant drop in protocol protocol checklist errorschecklist errors while while unintentional unintentional checklist errors have remained unchangedchecklist errors have remained unchanged

Threat & Error Management Model (Doc 9803)Threat & Error Management Model (Doc 9803)

CrewCrew ErrorError

 

InconsequentialInconsequential

ErrorError InducedInduced

IncidentIncident oror AccidentAccident

ThreatsThreats

ThreatThreat ManagementManagementThreatThreat InducedInduced

IncidentIncident oror AccidentAccident

UndesiredUndesired AircraftAircraft StateState

CrewCrew UndesiredUndesired

AircraftAircraft StateState

ResponsesResponses

CrewCrew ErrorError ResponsesResponses

TEM Model asTEM Model as Licensing tool (ICAO)Licensing tool (ICAO)Training toolTraining tool (Numerous airlines)(Numerous airlines)Safety management tool (IATA) Safety management tool (IATA) Research tool (Boeing)Research tool (Boeing)

The Expanding Role of the TEM ModelThe Expanding Role of the TEM Model

TEM - Licensing Tool (ICAO FCLT/P)TEM - Licensing Tool (ICAO FCLT/P) PerformancePerformance criteriacriteriaCompetenciesCompetencies SKA’sSKA’s

ErrorError ManagementManagement

Threat ManagementThreat Management

UndesiredUndesired AircraftAircraft StateState ManagementManagement

•FollowFollow SOPsSOPs•PerformPerform BriefingBriefing•StateState plansplans•AssignAssign workloadworkload•ContingencyContingency planningplanning•MonitorMonitor//CrossCross-check-check•ManageManage automationautomation•……•……•……

CRMCRM skillsskills

TEMTEM principlesprinciples

Managing the Red FlagsManaging the Red Flags

System resourcesSystem resources GPWSGPWSAutomationAutomationATCATCDispatchDispatch

SOP’s SOP’s ChecklistsChecklistsBriefingsBriefingsTrainingTraining……

Managing the Red Flags (cont.)Managing the Red Flags (cont.)

The human contributionThe human contribution

ProficiencyProficiency VigilanceVigilanceAssertivenessAssertivenessMonitoringMonitoring

DecisionDecision makingmakingExperienceExperienceLeadershipLeadershipDiscipline Discipline ……

Real World Human PerformanceReal World Human Performance

FlapsFlapsomittedomitted

EffectiveEffectivewarningwarning

NormalNormaloperationoperation

ChecklistChecklist worksworks

Descent, Approach & LandingDescent, Approach & Landing

TOTOD

1000010000 ft.

ConfigurationConfiguration

FAFFAF

Flare

LandingLanding

BottomBottom LinesLines

Most errors (LOSA)Most errors (LOSA)Most consequential errors (LOSA)Most consequential errors (LOSA)Most accidents worldwide (ALAR)Most accidents worldwide (ALAR)

Highest riskHighest riskphase of flightphase of flight

Briefing the ApproachBriefing the Approach

Crews who brief Crews who brief before before TOD commit TOD commit 1.6 times less 1.6 times less errorserrors during descent, approach and land than during descent, approach and land than crews who brief crews who brief afterafter TOD TOD

Landing runway changes occurred Landing runway changes occurred 8 percent8 percent of the of the timetimeTraining point: brief approach prior TODTraining point: brief approach prior TOD

TEM Model - Training ToolTEM Model - Training Tool

CRM, CRM, circacirca 2003 2003 Beyond behavioural stereotypesBeyond behavioural stereotypesOperational performanceOperational performanceDevelop and support threat and error management Develop and support threat and error management

competenciescompetencies

The bottom line is safe and efficient The bottom line is safe and efficient operational performance, operational performance, notnot CRMCRM

SMS Data TriangleSMS Data Triangle

Normal Normal Operations Operations

FDAFDA Voluntary Voluntary ReportingReporting

A A Casque BleuCasque Bleu Perspective Perspective

LOSA: a LOSA: a tooltool to capture systemic data to capture systemic data NOTECHS: a CRM evaluation templateNOTECHS: a CRM evaluation template LOSA: context and cognitionLOSA: context and cognition NOTECHS/CRM: behavioursNOTECHS/CRM: behaviours CRM: just one piece of the infinitely broader CRM: just one piece of the infinitely broader

information picture developed from LOSAinformation picture developed from LOSA A situation, generated by misperceptions and A situation, generated by misperceptions and misinformation, that must be put behindmisinformation, that must be put behind

LOSA, CRM and NOTECHSLOSA, CRM and NOTECHS