websat: a proactive system to capture aviation maintenance errors web-based surveillance &...

WebSAT:A Proactive System To Capture Aviation Maintenance Errors

Web-based Surveillance & Auditing Tool

Clemson WebSAT Team:Anand K. Gramopadhye

Joel S. Greenstein

Kunal KapoorNikhil Iyengar

Pallavi Dharwada

Sponsor: FAA

Industry Partner: FedEx, NWA and other

airlines

Human Computer Systems Laboratory

Department of Industrial Engineering

Clemson University, Clemson, SC 29634http://www.ces.clemson.edu/ie/research/hcsl/

websat/


Overview

• Background

• Motivation

• Problem Statement

• Research Overview

• WebSAT Research Objective

• Methodology

• Discussion and Conclusions

• Significant Milestones

• Dissemination

Slide 1


• The mission of FAA is to -

– Provide safe and reliable air transportation, and

– Ensure airworthiness of the aircraft

• Mission can be achieved by

– Improving the safety

– Minimizing aircraft accidents

Background

Slide 2


• Maintenance Error - a crucial factor in aircraft accidents

– 23% involved incorrect removal or installation of components

– 28% involved a manufacturer or vendor maintenance/inspection error

– 49% involved error due to an airline’s maintenance policy

– 49% involved poor design leading to maintenance errors

Motivation

(Source: Rankin et al., 2000)

Slide 3


Motivation Contd.

NTSB Identification: ATL01IA001 || Air Carrier: Air Carrier operation of Continental Airlines || Accident occurred October 01, 2000 in Birmingham, AL || Injuries: 1 Minor, 146 Uninjured. Aircraft: McDonnell Douglas MD-80, Reg: N69826

• During cruise flight, at flight level 310, an MD-80, operated by Continental Airlines experienced an electrical fire. An emergency was declared and the flight was diverted into Birmingham, Alabama, and landed without further incident. The examination of the airplane disclosed a 2 by 1 1/2 inch fire-damaged hole in the left jump seat wall.

• The failure of maintenance personnel to follow Fleet Campaign Directive (FCD) on how to install a certificate holder

Slide 4


Motivation Contd.

NTSB Identification: CHI03IA027 || Air Carrier: United Airlines Accident occurred November 21, 2002 in Chicago, IL || Injuries: 82 Uninjured. Aircraft: Airbus A319-131, Reg: N804UA

• United Airlines (UAL) flight 603 received minor damage when it landed on runway 04R (8,071 feet by 150 feet, asphalt) at the Chicago O'Hare International Airport. The airplane landed with the nose landing gear (NLG) wheels turned 90 degrees to the direction of travel. The 14 CFR Part 121 flight was being conducted in visual meteorological conditions and an instrument flight rules flight plan was filed.

• The maintenance facility improperly assembled and installed the nose landing gear shock absorber assembly. Factors were the improper assembly which allowed the nose gear to turn 90 degrees to the direction of travel.

Slide 5


Motivation Contd.

NTSB Identification: DCA00MA079. || Air Carrier: Air Carrier operation of AirTran Airlines Inc. || August 08, 2000 in Greensboro, NC || Injuries: 13 Minor, 50 Uninjured. Aircraft: Douglas DC-9-32, Reg: N838AT

• AirTran Airways flight 913 executed an emergency landing at Greensboro Piedmont-Triad International Airport (GSO) shortly after declaring an emergency due to an in-flight fire and smoke in the cockpit. An emergency evacuation was performed. Of the 58 passengers and 5 crewmembers on board, 3 crewmembers and 5 passengers received minor injuries from smoke inhalation. The airplane sustained substantial fire, heat, and smoke damage. Visual meteorological conditions prevailed at the time of the accident.

• A phase-to-phase arc in the left heat exchanger cooling fan relay, which ignited the surrounding wire insulation and other combustible materials within the electrical power center panel.

• Contributing to the left heat exchanger fan relay malfunction was the unauthorized repair that was not to the manufacturer's standards and the circuit breakers' failure to recognize an arc-fault.

Slide 6


HumansEquipment

Machines Environment

Maintenance System

Slide 7


• Airline companies ensure – Supervision over maintenance operations– Evaluation of internal and external factors– Adherence to quality assurance requirements and

FAA regulations• Airline companies oversee their

– Flight procedures– Operating methods– Airman qualifications– Auditors’ proficiency– Aircraft maintenance activities

Maintenance System

Slide 8


• Human is the central part of the aviation system

• Human factors research

– places emphasis on humans

– focuses on development of error tolerant systems

– aims at improved safety by evaluating aircraft maintenance activities

– conducts evaluation of maintenance activities in a rigorous fashion

Human Factors in Aviation

Slide 9


Methods to Track Maintenance Errors

• Some of the existing methods to minimize maintenance errors:

METHOD ADVANTAGE DISADVANTAGE

1. Error classification schemes: For example

HFACS maintenance extension taxonomy

Follow up errors using web-based information management system

• Reactive in nature

• Limited causality embedded in the schemes

2. Maintenance error decision aid (MEDA)

Helps identify contributing factors that led to an accident


• Dependent on the maintenance technician willingness to be interviewed about the error

3. Surveys and Questionnaires

Provide comprehensive description

• Lacks empirical data

Slide 10


Problem Statement

Northwest (Aug, 2000)


• Lack of standardization of data collection– Within an airline

– Across the industry

Cause of the accident: Crew failed to secure aircraft parking brake during

maintenance

Slide 11


• How to achieve an effective maintenance system?

– Seek input from diverse, reliable, and relevant sources

– Perform data collection, data reduction, and data analysis in a standardized fashion

– Generate trend analysis for causal factors within and across organizations

– Pro-actively identify factors contributing to maintenance errors

• Use the historical data to generate trends• Generate an overall safety index

Research Overview

Slide 12


WebSAT Research Objective

• Web-based Surveillance & Auditing Tool

– Capture data from aviation maintenance processes

– Analyze data obtained from the maintenance processes

– Perform risk assessment and generate trends using the data obtained

Slide 13


• Product Description

– A distributed application, incorporating a recommended categorization and data collection scheme for surveillance and auditing of maintenance operations

– A data reduction module that allows analysts to perform preliminary data analysis

– A data analysis module that facilitates trend analysis

WebSAT Mission Statement

Slide 14


• Key Business Goals

– Achieve standardized data collection, reduction and analysis of maintenance errors across geographically dispersed entities of the airline industry

– Develop a proactive system that captures maintenance errors

– Accomplish trend analysis in future versions of WebSAT

Mission Statement Contd.

Slide 15


• Primary Market

• Air Transportation Industry

• Assumptions

• Develop WebSAT such that it adheres to FAA standard research software design specifications (For example SQL server, ASP.NET, PHP)


Slide 16


• Stakeholders – FAA

– Carrier Quality Assurance Department including QA representatives and auditors

– Carrier Information Technology Department


Slide 17


• Task analytic & user centered software lifecycle methodology guides the following phases

– Phase I: Identify process measures

– Phase II: Design and develop WebSAT prototype

– Phase III: Develop data analysis module

Methodology

Slide 18


• Phase I: Identifying process measures

What are Process Measures?

– The process measures incorporate the response and observation-based data collected during surveillance, audits, and the airworthiness directives control processes

– Analysis will be conducted based on data obtained from process measures to identify the potential problematic areas affecting the safety of an aircraft

– The performance of processes, vendors, auditors and Quality Assurance

representatives will also be evaluated

Phase I

Slide 19


• Phase I: Identifying process measures– Discover

• Understand surveillance, auditing and AD work functions

• Conduct interviews, focus groups and observations– Identify

• Determine process measures

– Validate• Ensure measures are representative of those used

across maintenance facilities• Survey other stakeholders to obtain consensus on

process measures

Phase I Contd.

Slide 20


Phase I:Research Methodologies

• Research methodologies adopted in identifying the key maintenance oversight processes and their functionality

– Interviews

– Focus groups

– Observation sessions

– Studying documentation

– Questionnaires

(Source: Iyengar et al., 2004)

Slide 21


• Research methodologies (contd.)– Interviews

• Meeting airline QA managers• Understand the work environment• Collect useful documents• Stakeholders meet the research team

– Observation sessions• Understand how maintenance is done• Observe current product users in work environment

– Document studies• Highly regulated industry

– Questionnaires• Web-based process measures evaluation survey (remotely)

with partnering airlines

Slide 22

Phase I:Research Methodologies


Modules in WebSAT • These modules directly influence the air worthiness of the

aircraft

• The four key processes/modules in the QA department of FedEx identified for WebSAT are:– Surveillance: Oversight of day-to-day maintenance and inspection

activities on the airplane– Technical Audits: System level evaluation of standards and procedures of

suppliers, fuel vendors, and ramp operations done on a periodic basis– Internal Audits: Evaluation of internal processes in the departments of an

organization– Airworthiness Directives: Evaluation of the applicability, loading, and

tracking of airworthiness directives

Slide 23

Phase I:WebSAT Modules


• Surveillance

– Primary Objective: • Achieve planned and random sampling , an accurate, real-time,

and comprehensive evaluation of vendor’s compliance with the FedEx and FAA approved CAMP, GMM, and regulatory requirements

Slide 24



• Technical Audits

– Looks at the system and not the airplane (for example: compliance of flight procedures)

– A system is audited annually (longer period)

– Types• Supplier audits

• FMR audits

Slide 25



• Technical Audits

– Supplier Audits• Select audit standards and complete

the audit

• Report findings to manager

• Generate “corrective actions” report

– FMR Audits• Aircraft fueling, Line maintenance, Ramp

operations

• Oversight of these functions accomplished for FedEx owned or contracted facilities

Slide 26



• Internal Audits

– The evaluation of internal processes in the departments of an organization to verify their compliance with regulatory, company and departmental policies and procedures.

– Five work areas in Internal Audits at FedEx• EMM, FOD, SAI, EPI, and Ad hoc audits• EMM and FOD are termed as internal audits which together

comprise of approximately 45 audits (within a department)• SAI and EPI are internal evaluations that are part of ATOS (across

the departments)– work towards risk mitigation as opposed to corrective action

• Ad hoc audits are those which are conducted as a result of problems that shoot up unexpectedly.

Slide 27



• Airworthiness Directives Control Group:

– check for applicability of an AD

– responsible for the implementation of any new, revised or corrected ADs

– Apply AMOC or use ADNT depending on the AD applicability

Slide 28



WebSAT Framework

Tier-1

Tier-2

Tier-3

ANALYSIS

Risk (impact variables)

Non-risk variables

Categories

Process Measures 1. ___ 1. ___ 1. ___ 1. ___ 2. ___ 2. ___ 2. ___ 2. ___ 3. ___ 3. ___ 3. ___ 3. ___ …. …. …

DATA Surveillance Technical

Audits Airworthiness

directives

An

alysis

An

alysis

Internal Audits

Slide 29

Phase I:WebSAT Framework


• Existing number of process measures for surveillance: 17

– Data collected using work cards

• No process measures in place for Technical Audits

– Data collected by virtue of checklists

• Existing number of process measures for Internal Audits: 6

– Data collected by virtue of checklists

• No process measures in place for ADCG

– Data collected by virtue of checklists and canned review statements

Phase I:Process Measures

Slide 30

Identified List of Process Measures

WebSAT

Surveillance Auditing AD Control

Identified Process Measures

Technical Audits Internal Audits

Existing and Identified

Process Measures


Administration

Records

Manuals

Procedures

Safety

Training


Compliance/Documentation

Inspection

Facility Control

Training & Personnel

Procedures

Data Control

Safety

Loading & Tracking

Information Verification

No existing Process Measures

No existing Process Measures

Documentation Surveillance

Facility Surveillance

Procedures and Manual Violation

In Process

Verification

Final Aircraft Walk around

Slide 31

FROM

TO

Identification of Process Measures for Surveillance


• Difficulties associated in categorizing data from surveillance work function using the existing set of process measures

– Ambiguities exist in classifying data into process measures using CASE standards, GMM and IPM.

– Difficulty to choose the most appropriate one for each work card

– Memorizing process measures is not a primary task to the QA representatives

Slide 33

Phase I:Process Measures


• Online-survey for validation of identified process measures– A 2-phase online survey was conducted for the

validation of process measures

– First phase with FedEx

– Second phase with additional partnering airlines (data still awaited)

– Surveillance has 6 process measures

– Internal audit has 6 process measures

– Technical audits have 7 process measures

– Airworthiness directives group has 2 process measures

Slide 34

Phase I:Validation


• Online-survey for validation of identified process measures

Slide 35

Phase I:Validation Contd.


• Phase II: Develop the prototype of WebSAT

– Task analytic and user-centered software development methodology

• Converting customer statements to need statements

• Develop metrics to measure the performance of the tool with respect to the identified needs

• Competitive benchmarking to set target specifications for the tool (Help!)

Phase II

(Source: Ulrich et al., 2004)

Slide 36


• Sample Customer Statement- Needs- Metrics

Phase II:Sample metrics

Sr. Customer Statements Needs Metrics Units

1

The auditor is aware of a flag raised by the QA rep during a particular surveillance activity and can focus his audit on that area.

The tool allows the auditor to view discrepancies which occurred during surveillance.

Time taken by the auditor to view surveillance findings

Sec

2 Generate a standardized score for the vendor

The tool generates a standardized score for the vendor.

List of attributes to evaluate the vendor

List

Slide 37


• Interface design

– Develop user profiles

– Develop personas

– Develop scenarios

– Generate concepts

– Combine and refine concepts

– Develop low fidelity prototypes

– Test it with users

Phase II: Interface Design

Slide 38


• Work in Progress– Development of

Technical Audit Module Prototype

Phase II: Work in Progress

Slide 39


• Typical persona for technical audit

– Name: Eric Brandon– Age: 38– Job: Auditor at FedEx for the past 4 years– Work hours: 8 am to 5 pm– Education: B.S. Aeronautical Engineering– Location: Memphis, TN– Income: $ 90,000/yr– Skills: Received best auditor award consecutively

three times from FAA for the years 2000, 2001 and 2002. Worked in the maintenance hangar for 8 years and is in his current position based on past experience.

– Hobbies: Golf, football, parasailing, Jazz music and travel with family

Phase II: TA Persona

Slide 40


• Typical persona for technical audit contd.– Goals: Eric performs audits at various vendor

locations inside and outside the U.S. – Tasks:

• Eric uses a checklist to evaluate a vendor.

• Eric enters the responses to the questions in the check-list.

• Eric enters comments in the audit report based on the overall performance of the vendor.

• Eric also reports the corrective actions that need to be addressed by the vendor.

Phase II: TA Persona Contd.

Slide 41


– Expectations: • Eric thinks that the product is useful to him

because it helps him analyze the data better.

• He thinks that the product is easy to use and will help him to evaluate the vendors in a standardized fashion.

Slide 42

Phase II: TA Persona Contd.


• Typical scenario for technical audit – Positive Scenario : Eric now wants to enter audit data which

includes the corrective actions recommended and his overall comments on the audit. He logs into the system, finds the appropriate audit into which the data needs to be entered. After entering the data, he generates the audit report and sends it to his manager, William Cox.

– Negative Scenario : Eric had entered preliminary audit data into the system. He continues with his audit on shop floor for a couple of days and returns to the system. He logs into the system but he is apprehensive if the system has stored the information on audit he did not complete. Alas! He could not find the appropriate audit into which the data was entered. He again started entering the entire audit information in the new file.

Slide 43

Phase II: TA Scenario


• Typical persona for Surveillance

– Name: Bob Lewis– Age: 42 years– Job: Surveillance Rep. at FedEx for

the past 2 years– Work hours: 7 am to 4 pm– Location: Mobile, AL– Income: $ 90,000/yr– Skills: Worked in the maintenance hangar for 3 years in another airline.

Has a keen eye for detecting maintenance problems and is in his current position as QA based on past experience.

– Hobbies: football, and reading– Goals: Bob performs surveillance at Mobile Aerospace Engineering, AL

and believes that his primary motto is to see that the flights which leave the hangar are airworthy.

Slide 44

Phase II: Surveillance Persona


• Typical persona for Surveillance contd.– Tasks:

• Bob uses the surveillance system to identify incoming aircrafts.

• Bob performs surveillance on the work cards that are chosen by the system through random sampling.

• Bob enters results of the surveillance into system. • Bob also enters non-routines that need to be attended by the vendor.

– Expectations: • Bob feels that the tool would make the categorization of data easy but it

adds a step to his routine life. • Bob thinks that the product is useful to him because it helps him analyze the

data better. • He thinks that the product will help him to evaluate the vendors in a

standardized fashion.

Slide 45

Phase II: Surveillance Persona Contd.


• Phase III

– Develop Advanced Data Analysis Module into Prototype Application

• module will enable the analyst to conduct advanced analysis of selected data sets

• identify problem areas forming the first step to conducting risk assessments

Phase III

Slide 46


WebSAT Framework

Phase III: Data Format

Tier-1

Tier-2

Tier-3

ANALYSIS

Risk (impact variables)

Non-risk variables

Categories

Process Measures 1. ___ 1. ___ 1. ___ 1. ___ 2. ___ 2. ___ 2. ___ 2. ___ 3. ___ 3. ___ 3. ___ 3. ___ …. …. …

DATA Surveillance Technical

Audits Airworthiness

directives

An

alysis

An

alysis

Internal Audits

Format of data collected:

1. Surveillance: Accepts, rejects and non-routine for a work card

2. Technical and Internal Audit: Yes/No, Multiple choice and open ended responses to questions asked in an audit

3. Airworthiness Directives: Status and location of an AD or EO

Slide 47


• Vision

– The goal of WebSAT is to allow the user to understand the implication of the data collected and presented- not only to the user such as auditor or surveillance representative but also to the managers

– WebSAT risk model aims at presenting the risk present in terms of process measures, impact variables and safety index

Phase III: Vision

Slide 48


Phase III:Data Analysis

Slide 49

Web-based Surveillance & Auditing Tool Slide 43

Phase III:Data Analysis Contd.

Administration

Question 1

Question 2

Does the department have an active self audit program with properly documented audits and corrective actions?

Are aircraft parts properly stored, identified and have traceability?

Procedures


Discussion & Conclusion

• WebSAT intends to

– enhance the utility of surveillance, auditing and airworthiness directive activities associated with commercial aircraft maintenance by

• Identification of maintenance related process measures and related safety indices

• Identification of potential problems at geographically dispersed maintenance sites through data standardization

• Promotion of accessible display of trends in data

• Assumptions

– It assumes integrity of data

– It evaluates only those who directly interact with it. Hence, it does not evaluate the maintenance personnel who work on the aircraft

– It provides access to ATOS but does not include ATOS data for its analysis or vice-versa

Slide 51


Significant Milestones Ahead

1. Trip to other participating airlines

2. Validation of identified process measures

3. Design of the technical audit prototype

• Screen Designs

• Competitive benchmarking

• User testing

4. Identification of impact variables

Slide 52


Dissemination

• Dharwada, P., Iyengar, N., Kapoor, K., Gramopadhye, A.K., and Greenstein, J.S., Web-Based Surveillance and Auditing Tool (WebSAT): A Proactive System to Capture Maintenance Errors, Proceedings of Safety Across High-Consequence Industries, St. Louis, Missouri, March 2004.

• Dharwada, P., Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Standardized Auditing and Surveillance of the Aircraft Maintenance Operations, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.

• Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Selection of Data Gathering Methodologies for the Aviation Maintenance Industry, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., A Strategy for the Development of a Web-based Tool to Reduce Aviation Maintenance Errors, Proceedings of Human Factors and Ergonomics Society, New Orleans, September 2004.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy for Evaluation of Aircraft Maintenance Operations Using Process Measures, International Symposium of Aviation Psychology, Oklahoma City, April 2005.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy to Identify Process Measures for Surveillance in Aviation, Proceedings of the Industrial Engineering Research Conference, Atlanta, May 2005.

• Iyengar, N., Kapoor, K., Dharwada, P., Greenstein, J. S., and Gramopadhye, A. K., WebSAT: Development of Process Measures for Aircraft Safety, International Journal of Applied Aviation Studies, 2005 (In review).

Slide 53


Thank You

websat: a proactive system to capture aviation maintenance errors web-based surveillance &...

Documents

airlines maintenance

certificate holder slide

cruise flight

flight level

chi03ia027 air carrier

atl01ia001 air carrier

aircraft mission

united airlines accident