deriving bayesian classiﬁers from flight data to enhance ... · a reference model that provides a...

Annual Conference of the Prognostics and Health Management Society, 2011

Deriving Bayesian Classifiers from Flight Data to EnhanceAircraft Diagnosis Models

Daniel L.C. Mack 1, Gautam Biswas 1, Xenofon D. Koutsoukos 1, Dinkar Mylaraswamy 2, and George Hadden 2

1 Vanderbilt University, Nashville, TN, 37203, [email protected]

[email protected]@vanderbilt.edu

2 Honeywell Aerospace, Golden Valley, MN 55422, [email protected]

[email protected]

ABSTRACTOnline fault diagnosis is critical for detecting the on-

set and hence the mitigation of adverse events that arisein complex systems, such as aircraft and industrial pro-cesses. A typical fault diagnosis system consists of: (1)a reference model that provides a mathematical repre-sentation for various diagnostic monitors that providepartial evidence towards active failure modes, and (2)a reasoning algorithm that combines set-covering andprobabilistic computation to establish fault candidatesand their rankings. However, for complex systems ref-erence models are typically incomplete, and simplify-ing assumptions are made to make the reasoning algo-rithms tractable. Incompleteness in the reference mod-els can take several forms, such as absence of discrim-inating evidence, and errors and incompleteness in themapping between evidence and failure modes. Inaccu-racies in the reasoning algorithm arise from the use ofsimplified noise models and independence assumptionsabout the evidence and the faults. Recently, data miningapproaches have been proposed to help mitigate someof the problems with the reference models and reason-ing schemes. This paper describes a Tree AugmentedNaı̈ve Bayesian Classifier (TAN) that forms the basisfor systematically extending aircraft diagnosis referencemodels using flight data from systems operating withand without faults. The performance of the TAN mod-els is investigated by comparing them against an expertsupplied reference model. The results demonstrate thatthe generated TAN structures can be used by human ex-perts to identify improvements to the reference model,by adding (1) new causal links that relate evidence tofaults, and different pieces of evidence, and (2) updatedthresholds and new monitors that facilitate the derivationof more precise evidence from the sensor data. A casestudy shows that this improves overall reasoner perfor-mance.

This is an open-access article distributed under the terms ofthe Creative Commons Attribution 3.0 United States License,which permits unrestricted use, distribution, and reproductionin any medium, provided the original author and source arecredited.

1. INTRODUCTIONAn important challenge facing aviation safety is early de-tection and mitigation of adverse events caused by sys-tem or component failures. Take an aircraft, which con-sists of several subsystems such as propulsion, avionics,bleed, flight control, and electrical; each of these subsys-tems consists of several dozen interacting componentswithin and between subsystems. Faults can arise in oneor more aircraft subsystems; their effects in one systemmay propagate to other subsystems, and faults may in-teract. To detect these faults, an onboard fault diagnosissolution must be able to deal with these interactions andprovide an accurate diagnostic and prognostic state forthe aircraft with minimal ambiguity.

The current state of online fault diagnosis is focusedon installing a variety of sensors onboard an aircraftalong with reasoning software to automatically interpretthe evidence generated by them, and infer the presenceof faults. One such state of the art system is the AircraftDiagnostic and Maintenance System ADMS (Spitzer,2007) that is used on the Boeing B777. ADMS canbe broadly categorized as a model-based diagnoser thatseparates system-specific knowledge and the inferencingmechanism.

Consider characteristics of some typical faults arisingin aircraft subsystems. Turbine blade erosion is a naturalpart of turbine aging and wearing of the protective coat-ing due to microscopic carbon particles exiting the com-bustion chamber. As the erosion progresses over time,it starts to affect the ability of the turbine to extract me-chanical energy from the hot expanding gases. Even-tually this fault manifests itself as increase in fuel flowand gradual degradation of engine performance. Thiscausal propagation of faults is usually known to a do-main expert and captured mathematically using a staticsystem reference model. As evidence gets generated byaircraft installed sensors, a reasoning algorithm “walks”the relevant causal paths and infers the current state ofthe aircraft—in this case, turbine erosion of the propul-sion engine.

The ADMS uses a fault propagation system refer-ence model that captures the interactions between air-craft components under various operating modes. ABayesian belief propagation network together with the

1


Bayesian update rule provides an ideal framework foronboard diagnostic reasoning using the reference model.It provides the necessary transparency for certification asa safety system, while allowing the subsystem manufac-turer to encode proprietary fault models. The generationof this reference model is mostly a manual process, andoften the most tedious step in the practical developmentand deployment of an ADMS. While most of the knowl-edge about fault propagation can be derived from earlieraircraft designs, upgrades to component design (for ex-ample using active surge control rather than passive on-off surge prevention) create gaps in the knowledge base.As the engineering teams “discover” the new knowledgefrom an operating fleet, they are translated into expertheuristics that are added to the specific aircraft model,rather than applying systematic upgrades to the overallreference model that was generated at design time.

Many of the shortcomings of the ADMS can be at-tributed to incomplete and incorrect information in thesystem reference model. In other words, there is a miss-ing link in making systematic upgrades and incrementsto the reference model as vast amount of operational datais collected by operating airlines. We look at this prob-lem as a “causal structure discovery” problem. Specifi-cally, learning causal structures in the form of a BayesianNetwork built for classical fault diagnosis, wherein thenodes represent system faults and failures (causes) andavailable diagnostic evidence (symptoms)(Pearl, 1988).Unlike associations, Bayesian networks can be used tobetter capture the dependencies among failures (failurecascade from one subsystem to another) and evidencecascade (failure mode in one system triggering a symp-tom in a nearby component). We adopt this approach,and develop a data mining approach to updating existingreference models with new causal information.

This paper presents a case study, an adverse event sur-rounding an in-flight shutdown of an engine, which wasused to systematically augment an existing ADMS refer-ence model. Section 2. describes the basic principles andthe constituents of a model-based onboard fault reasoner.Section 3. describes the problem statement that formallydefines the model augmentation to be derived using op-erational data. Next, section 4. describes the availablehistoric data surrounding the adverse event. Section5. briefly discusses the challenges in taking operationaldata and transforming it into a form that can be used bythe data mining algorithms. Section 6. then discussesthese data mining algorithms employed for construct-ing the diagnostic classifiers as Tree-Augmented Naı̈veBayesian Networks (TANs). Section 7. presents exper-imental results of this case study to show how a humanexpert could utilize the classifier structure derived fromflight data to improve a reference model. Metrics are de-fined for evaluating classifier performance, and a num-ber of different experiments are run to determine whenimprovements can be made in the existing model. Sec-tion 8. presents a summary of the approach, and outlinesdirections for future work for diagnostic and prognosticreasoning using the data mining algorithms.

2. BACKGROUND ON REFERENCE MODELSAND REASONERS

Model-based strategies that separate system-specificknowledge and the inferencing mechanism are preferred

for diagnosing large, complex, real-world systems. Anaircraft is no exception to this, as individual componentsuppliers provide system-specific knowledge that can berepresented as a bipartite graph consisting of two typesof nodes: failure modes and evidence. Since this knowl-edge acts as a baseline for diagnostic inferencing, theterm “reference model” is also used to describe this in-formation. The set F captures all distinct failure modesdefined or enumerated for the system under considera-tion. A failure mode fmi ∈ F may be occurring or notoccurring in the system, which is indicated by a 1 (occur-ring) or 0 (not occurring) state. Often a −1 an unknownstate is also included in the initial state description. Thefollowing are shorthand notations regarding these asser-tions.

fmi = 0 ⇔The failure mode is not occurringfmi = 1 ⇔The failure mode is occurring (1)

Every failure mode has an a priori probability of oc-curring in the system. This probability is given byP (fmi = 1). Failure modes are assumed to be inde-pendent of one another, i.e., given any two failure modesfmk and fmj , P (fmk = 1|fmj = 1) = P (fmk = 1).

To isolate and disambiguate the failure modes, com-ponent suppliers also define an entity called “evidence”that is linked to sensors and monitors in the system. Theset DM denotes all distinct diagnostic monitors definedfor the system under consideration. A diagnostic monitorassociated with mj ∈ DM , can either indict or exoner-ate a subset of failure modes called its ambiguity group.In other words, each monitor mi in the system is labeledby three mutually exclusive values allowing a monitor toexpress indicting, exonerating or unknown support forthe failure modes in its ambiguity group. The notationsare described in equation (2).

mi = 0 ⇔ Exonerating evidencemi = 1 ⇔ Indicting evidence

mi = −1 ⇔ Unknown evidence(2)

An ideal monitor mj fires only when one or more fail-ure modes in its ambiguity group are occurring. Giventhe fact that the ith failure mode is occurring in the sys-tem, dji denotes the probability that monitor mj willprovide indicting evidence under this condition.

dji ⇔ P (mj = 1|fmi = 1), (3)

dji is called the detection probability of the jth mon-itor with respect to failure mode fmi. A monitor mayfire when none of the failure modes in its indicting setare present in the system. False alarm probability is theprobability that an indicting monitor fires when its cor-responding failure modes in its ambiguity group are notpresent in the system. That is,

εj ⇔ P (mj = 1|fmi = 0,∀fmi ∈ Ambiguity Set)(4)

Designing a monitor often requires deep domainknowledge about the component or subsystem, but thedetails of this information may not be important from thereasoner’s viewpoint. A more abstract view of the moni-tor is employed in the reasoning algorithm. This abstrac-tion is shown in Figure 1. With few exceptions, most

2


Figure 1: Abstraction of Diagnostic monitor

diagnostic monitors are derived by applying a thresholdto a time-series signal. This signal can be a raw sensorvalue or a derived quantity from a set of one or moresensor values. We call this a condition indicator (CI)and denote it as x(t). Assuming a pre-defined thresholdvalue θ, we set m = 1 ⇔ x(t) ≶ θ. A diagnostic mon-itor may specify the underlying condition indicator andthe threshold or simply provide the net result of applyinga hidden threshold.

Figure 2 illustrates an example reference model graph-ically, with fault modes (hypotheses) as nodes on the left,and diagnostic monitors mi ∈ DM on the right. Eachlink has an associated detection probability, i.e., condi-tional probability P (mj = 1|fmi = 1). In addition,fault nodes on the left contain the a priori probabilityof fault occurrence, i.e., P (fmi). Probabilities on theDM nodes indicate the likelihood that a particular moni-tor would indicate a fault in a nominal system. Bayesianmethods are employed to combine the evidence providedby multiple monitors to estimate the most likely faultcandidates.

The reasoner algorithm (called the W-algorithm) com-bines an abductive reasoning scheme with a forwardpropagation algorithm to generate and rank possible fail-ure modes. This algorithm operates in two steps: (1) Ab-ductive reasoning step: Whenever a diagnostic monitorm1 fires, it provides either indicting (if m1 = 1) or exon-erating (if m1 = 0) evidence for the failure modes in itsambiguity set, AG = {fm1, fm2, . . . fmk}. This stepassumes that the firing of a DM implies at least one of thefaults in the ambiguity set has occurred; and (2) Forwardreasoning step: For each fmi belonging to AG, this stepcalculates all other diagnostic monitors that may fire ifany of the failure modes are indeed occurring. Theseare called the evidence of interest. Let m2,m3, · · · de-note this evidence of interest set. Some of these moni-tors may be indicting evidence, for example m2 = 1 orthey may be exonerating evidence, for example m3 = 0.The reasoning algorithm calculates the joint probabilityP (fm1 = 1,m1 = 1,m2 = 1,m3 = 0, . . . ) of a spe-cific failure mode fm1 occurring in the system. As ad-ditional monitors fire, the numeric values of these prob-abilities increase or decrease, till a specific failure mode

hypothesis emerges as the highest-ranked or the mostlikely hypothesis. The reasoning algorithm can gener-ate multiple single fault hypotheses, each hypothesis as-serting the occurrence of exactly one failure mode in thesystem.

The reasoning algorithm may not succeed in reduc-ing the ambiguity group to a single fault element. Thiscan happen for various reasons: (1) incompleteness anderrors in the reference model; (2) simplifying assump-tions in the reasoning algorithm; and (3) missing ev-idence (monitors) that support or discriminate amongfault modes. For example, a modest aircraft has over5000 monitors and failure modes; estimating the detec-tion probabilities, dji, even for this aircraft is a challeng-ing offline design task. Errors in dji, and more specif-ically missing the link between a monitor and a failuremode (incompleteness) can adversely affect the reasonerperformance. Further, to keep things simple for the rea-soner, a modeler may assume that the firing events formonitors are independent. This eliminates the need forthe modeler to provide joint probability values of theform, P (mj = 1,mk = 1|fmi = 1) (say for 4000 mon-itors and 1000 faults the modeler would have to provide1.56×1010 probability values), and instead approximateit as P (mj = 1|fmi = 1) × P (mk = 1|fmi = 1).This reduces the total number of probabilities to 4×106,which is still a large number but orders of magnitudeless than the number required for the joint distributionsand the order of the joint distributions grow exponen-tially when additional monitors fire. Designing a goodset of monitors is yet another challenging task. For ex-ample, the modeler may have overlooked a new monitormp that could have differentiated between failure modesfm1 and fm2.

Given the complexity of large systems such as anaircraft, incompleteness in the reference model is ex-pected. As one collects enough operational data, someof these gaps can be addressed. The collection of as-sumptions made about the fault hypotheses and moni-tors results in the probability update function for eachfault hypothesis, fmi∀i ∈ F , being computed usinga Naı̈ve Bayes model, i.e., P (fmi|mj ,mk,ml · · · =α × P (mj ,mk,ml · · · |fmi) = α × P (mj |fmi) ×

3


Figure 2: Example Reference Model

P (mj |fmi) × P (mj |fmi) × · · · where α is a normal-izing constant. The direct correspondence between thereference model and the simple Bayesian structure pro-vides opportunities to use a class of generative Bayesianmodel algorithms to build structures that are relevant fordiagnostic reasoning from data. These newly learnedstructures can then be used with a systematic approachfor updating the system reference model. The followingwork focuses on this systematic approach.

3. PROBLEM STATEMENTThe combination of the reference model and reasonerwhen viewed as a single fault diagnoser can be in-terpreted as a Noisy-OR classifier, which is a simpli-fied form of a standard Bayesian Network. These net-works that model diagnostic information (i.e., monitor-fault relations) can be built from data itself as a prac-tical application of data mining. A number of Ma-chine Learning techniques for building Bayesian net-works from data have been reported in the litera-ture (Friedman, Geiger, & Goldszmidt, 1997), (Cheng,Greiner, Kelly, Bell, & Liu, 2002),(Grossman & Domin-gos, 2004). For example, state-based hidden MarkovModels (HMMs) (Smyth, 1994) and even more gen-eral Dynamic Bayesian Network (DBN) (Dearden &Clancy, 2001), (Lerner, Parr, Koller, & Biswas, 2000),(Roychoudhury, Biswas, & Koutsoukos, 2008), (Verma,Gordon, Simmons, & Thrun, 2004) formulations canbe employed to capture the dynamics of aircraft behav-ior and effects of faults on system behavior and perfor-mance. However, rather than addressing the problem asa traditional data mining problem, it is approached as an

application that works to extend an existing ADMS. Inother words, the output of the data mining algorithmshave to be designed to provide information that sup-plements existing expert-generated reference models, asopposed to providing replacement forms of the refer-ence model with corresponding reasoner structures. Thistechnique can be construed as a method for supportinghuman experts, by having a human in the loop to in-terpret the findings generated by the data mining algo-rithm and make decisions on how to modify and updatethe existing reference model and reasoner structures. Byincluding humans, who verify and integrate the modelenhancements, we turn the verification into a straightfor-ward task for the human to either approve the selectedchanges, or ignore them.

A systematic approach to the data mining task in thiscontext is to discover elements of the reference modelthat can be augmented by comparing the results of a datadriven model produced from using a learning algorithmagainst an existing structure extracted from the referencemodel. The extraction of this existing structure from theReference Model begins by isolating a specific failuremode. The failure mode chosen is often guided by theavailable data where the mode was active. Isolating asingle failure mode from the reference model and themonitors that indict the mode produces a tree structurewhere the class node describes the binary presence ofthat fault. The indicators(the leaves of the tree) haveprobabilities for the indictment of the mode and falsealarm rates from the reference model that can be usedto construct the complete probabilistic space. This struc-ture and probabilistic information is the classical defini-

4


tion of a Naı̈ve Bayes classifier. With data and the useof algorithms to build a Bayesian structure, the modelcan be leveraged to improve this very specific structure.Limiting it to a structure that preserves the general treeof the Naı̈ve Bayes classifier eases the transition of in-formation from the learned structure back to the originalmodel. This is balanced with our desire to add limitedcausality into the network to help the expert understandif there are health indicators which are correlated. Thisinformation could be added back in limited ways to theisolated structure, without requiring the entire referencemodel to be changed from the Noisy-OR model. Thestructure that is chosen for learning is a Tree AugmentedNaı̈ve Bayesian network, which we will discuss in moredetail in section 6.

The information added back to the reference modelfalls into three areas:

1. Update Monitors Update the threshold θ associ-ated with a diagnostic monitor. The idea is to makethe monitor i more sensitive to failure mode j (sothat it can be detected earlier, if it is present) with-out sacrificing the false alarm rate, εj for the moni-tor.

2. Add Monitors to Failure Mode Update the iso-lated structure by adding a monitor that helps indictthe failure mode. Specifically this could take twoforms: (a) creating a new monitor with the requisiteprobabilistic information, and adding a new dji toassociate it with the failure mode, and (b) assigninga non-zero number for dji if the link did not alreadyexist with a previously created monitor.

3. Create Super Monitors Creating new monitorsthat combine existing monitors, say mi and mjsuch that the combination of monitor indictmentsasserts stronger evidence for a specific failure modefmk. That is, calculate a stronger value forP (mi = 1,mj = 1|fmk = 1) which is greaterthan P (mi = 1|fmk = 1)×P (mj = 1|fmk = 1).

In addition to establishing areas of improvementfound by comparing the data mining results with the ref-erence model, the computational complexity of the datamining algorithms should be manageable, so that theycan be used as exploratory analysis tools by the domainexperts. Potentially, the experts may apply a successiverefinement process by requesting a number of experi-mental runs, each using a specific data set from an op-erating fleet of aircraft, and the results from the nth ex-periment augments or confirms the reference model fromthe (n − 1)th experiment. This will result in a continu-ous learning loop wherein historical observations fromthe fleet are analyzed systematically to understand thecausal relations between failure modes and their mani-festations (monitors). In addition, building models fromthe data may also reveal unknown (or currently unmod-eled) dependencies among failure modes that are linkedto the adverse event situations under consideration. Overtime, this learning loop will increase the accuracy andtime to detection (while reducing false positives) in thediagnostic reasoner.

The next step is to explore and pre-process the avail-able aircraft flight data for the data mining task. The pre-processing plays a major role in determining the natureof information derived from the data, and, using prior

knowledge of the aircraft domain the pre-processing al-gorithms can be tailored to avoid the “needle-in-the-haystack” search problem.

4. AIRCRAFT FLIGHT DATAIt is important to extract flight data of the right type andform that will potentially help to find and validate newdiagnostic information. Since the goal is early and reli-able detection of an evolving fault in the aircraft, it is im-portant that the data set formed for analysis span severalcontiguous flights. This set should also include multipleaircraft to account for aircraft-to-aircraft variations andthe heterogeneity of flight conditions and flight paths.Our data set comes from a fleet of aircraft belonging to aregional airline from North America. The fleet consistedof 30+ identical four engine aircraft, each operating 2–5flights each day. This work examines data spanning threeyears of the airline’s operations.

The Aircraft Condition Monitoring System (ACMS)is an airborne system that collects data to support faultanalysis and maintenance. The Digital ACMS Recorder(DAR) records airplane information onto a magnetic tape(or optical) device that is external to the ACMS. Thisdata is typically stored in raw, uncompressed form. TheDAR can record information at a maximum rate of 51212-bit words per second via a serial data stream modu-lated in either Harvard Bi-Phase or Bi-Polar Return-to-Zero code. The recorded data is then saved permanentlyto a compact flash card. The ACMS can be programmedto record parameter data from the propulsion subsystem,the airframe, the aircraft bleed subsystem, and the flightmanagement system at a maximum rate of 16 Hz. Weapply our initial data retrieval and pre-processing algo-rithms to this raw time-series data that was made avail-able to us in the form of multiple CDs.

A second source of information we referenced for thisstudy was adverse event annotations that is available ina FAA developed Aviation Safety Information Analy-sis and Sharing (ASIAS) database system. The ASIASdatabase is a collection of adverse events reported by var-ious airline operators. On searching this database for thetime period of the flight data available to us revealed thatan engine shutdown event had occurred for one of the air-craft in our list. On one of the flights of this aircraft, thethird engine(out of four) aboard the aircraft shutdown au-tomatically. As a result, the flight crew declared an emer-gency situation and returned back to the airport wherethe flight originated. Fortunately, there were no casu-alties or serious injuries. For this study, we decided tofocus on this failure mode, mainly because the on boardreasoner or the mechanics who serviced the aircraft wereunable to detect any anomalies in the engine till the ad-verse event occurred.

In more detail, an adverse event such as an engineshutdown typically evolves as a sequence of anomalousevents and eventually leads to a situation, such as overheating, that causes the shutdown. For this case study,our objective was to analyze the ACMS flight data fromthe aircraft prior to adverse event with the goal of defin-ing anomalies in the system monitors that were not de-fined for the existing ADMS. The primary intent was touse these anomalous monitor reports to detect and iso-late the root cause for the failure as early as possible,so that the onset of the adverse event could be avoided.

5


Investigation of the airline maintenance crew reports af-ter the adverse event revealed that the root cause forthis adverse event was a faulty fuel metering hydro-mechanical unit(Fuel HMA) in the third engine, whichwas the engine that shut down. The fuel metering unitis a controller-actuator that meters fuel into the com-bustion chamber of the engine to produce the desiredthrust. Given that we now knew the time of the adverseevent and the root cause for the particular engine failure,knowledge of the fuel metering unit impliedthat this wasa slowly evolving (i.e., incipient) fault that could verylikely start manifesting about 50 flights before the actualengine shutdown adverse event. Therefore, we extractedthe [−50, 0] flight interval for the analysis, where 0 indi-cates the flight number for which the adverse event oc-curred and −50 indicates 50 flights before this one. Weassumed that the particular aircraft under study had onefaulty and three nominal engines. We collected relevantdata (discussed below) for all of the engines, and then ranBayesian classifiers to discover the differences betweenthe faulty and the nominal engines.

As we discussed earlier, all of the aircraft for the re-gional airline were equipped with ADMS. The diagnoserreceives information at pre-defined rates from the di-agnostic monitors. In this case study, we also assumethat we had access to the sequence of condition indica-tors(CI) values that were generated. As discussed earlier,the binary monitor output is produced by applying a pre-defined threshold to the CI values. In our data mininganalysis, we use the CI’s as features, and compare thethresholds derived by the classifier algorithms against thethresholds defined in the reference model. The followingcondition indicators and diagnostic monitors were avail-able from this aircraft flight dataset.StartTime This CI provides the time the engine takes to

reach its idling speed. Appropriate threshold gener-ates the no start diagnostic monitor.

IdleSpeed This CI provides the steady state idlingspeed. Appropriate threshold generates the hungstart diagnostic monitor.

peakEGTC This CI provides the peak exhaust gas tem-perature within an engine start-stop cycle. Appro-priate threshold generates the overtemp diagnosticmonitor.

N2atPeak This CI provides the speed of the enginewhen the exhaust gas temperature achieves its peakvalue. Appropriate threshold generates the over-speed diagnostic monitor.

timeAtPeak This CI provides the dwell time when theexhaust gas temperature was at its peak value. Ap-propriate threshold generates the overtemp diagnos-tic monitor.

Liteoff This CI provides the time duration when the en-gine attained stoichiometry and auto-combustion.Appropriate threshold generates the no lightoff di-agnostic monitor.

prelitEGTC This CI provides the engine combustionchamber temperature before the engine attained sto-ichiometry. Appropriate threshold generates the hotstart diagnostic monitor.

phaseTWO This CI provides the time duration whenthe engine controller changed the fuel set-point

schedule. There are no diagnostic monitors definedfor this CI.

tkoN1, tkoN2, tkoEGT, tkoT1, tkoPALT These CIsprovide the fan speed, engine speed, exhaust gastemperature, inlet temperature and pressure alti-tude, respectively, averaged over the time intervalwhen aircraft is operating under take off conditions.There are no diagnostic monitors defined for theseCIs.

tkoMargin This CI provides the temperature margin forthe engine during take off conditions. Appropriatethreshold generates the medium yellow and low reddiagnostic monitors.

Rolltime This CI provides the time duration of the en-gine’s roll down phase. Appropriate threshold gen-erates the abrupt roll diagnostic monitor.

resdTemp These CI provide the engine exhaust gastemperature at the end of the engine’s roll downphase. Appropriate threshold generates the highrtemp diagnostic monitor.

N2atDip, dipEGTC These CIs provide the enginespeed and the exhaust gas temperature at thehalfway point in the engine’s roll down phase.There are no diagnostic monitors defined for theseCI.

N2cutoff These CI provide the rate of change of the en-gine speed at the halfway point in the engine’s rolldown phase. There are no diagnostic monitors de-fined for these CI.

This large volume of CI data (multiple aircraft, mul-tiple flights) provides opportunities to study aircraft en-gines in different operating scenarios in great depth anddetail. However, the data as extracted from the raw flightdata DAR files was not in a form that could be directlyprocessed by our classification algorithms. We had to de-velop data curation methods to generate the data sets thatcould be analyzed by the machine learning algorithms.

5. DATA CURATIONAn important requirement for the success of data driventechniques for knowledge discovery is the need to haverelevant and well-organized data. In our study, well-organized implied getting rid of unwanted details, be-ing able to structure the data on a timeline (having all ofthe CI’s aligned in time, and the monitor output insertedinto the time line as a sequence of events), and applyingfiltering algorithms to the noisy sensor data. Relevanceis an important concept, since it is necessary to extractsequences of data that contain information about the par-ticular situation being modeled. For example, if the goalis to design a classifier that can identify a faulty situationfrom one in which there is no fault, it is important that theclassifier be provided with both nominal and faulty data,so that it can derive the discriminating features from thedata. Further, the systems under study are complex, andthey operate in different modes and under different cir-cumstances. This information is likely to be importantfor the classification task, so the data needs to be appro-priately annotated with this information. It is clear thatunreliable data is unlikely to provide useful informationto an already effective reference model. Our (and oth-ers) experiences show that the data curation task is often

6


more time consuming and sometimes quite difficult (be-cause of noisy, missing, and unorganized data) as com-pared to the data mining task, which involved running aclassifier or a clustering algorithm on the curated data. Agood understanding of the nature of the data and how itwas acquired is critical to the success of the data miningtask.

In our study, the DAR files represented single flightsencoded in binary format. As a first step, we orga-nized several thousands of these files by the aircraft tailnumber. For each aircraft, the data was then organizedchronologically using the time stamp associated with theparticular flight. Since the case study involves an engineshutdown situation, the data was further classified basedon the engine serial number so that the data associatedwith each engine could be easily identified.

For practical reasons, given the size of the data, andthe need to extract specific sub-sequences for the datamining task, we designed a relational database to cre-ate an organized representation for the formatted data.This made it easier to access the relevant data for dif-ferent experimental analyses. For a data analysis ses-sion, step 1 involved formulating data base queries andcollecting the extracted data segments into the form re-quired by the classification algorithm. Step 2 was thedata curation step. A primary task at this step was re-moving all extraneous non-flight data. For this analy-sis, all ground-test information (data generated when themaintenance crew ran a test when the aircraft was on theground) was defined as anomalous and removed duringthe cleansing step. Step 3 involved running the classifi-cation algorithms.

6. TREE AUGMENTED NAIVE BAYESIANNETWORKS

The choice of the data driven techniques to apply to par-ticular problems is very much a function of the natureof the data and the problem(s) to be addressed usingthe data. The extracted portion of the reference modeldiscussed earlier can be modeled as a Naı̈ve Bayes rea-soner. The independence assumptions of the model mayalso be systematically relaxed to capture more discrimi-natory evidence for diagnosis. There are several inter-esting alternatives, but one that fits well with the iso-lated structure is the Tree Augmented Naı̈ve BayesianMethod (Friedman et al., 1997) abbreviated as the TANalgorithm. The TAN network is a simple extension tothe Naı̈ve Bayes network formulation. The Root (thefault mode) also know as the class node is causally re-lated to every evidence node. In addition, the indepen-dence assumption for evidence nodes is relaxed. An evi-dence node can have at most two parents: one is the classnode, the other can be a causal connection to anotherevidence node. These constraints maintain the directedacyclic graph requirements and produce a more nuancedtree that captures additional relationships among the sys-tem sensors and monitors. At the same time, the learningalgorithm to generate the parameters of this structure iscomputationally simpler than learning a general Bayesnet structure.

The TAN Structure can be generated in several dif-ferent ways. One approach uses a greedy search thatconstrains the graph from building “illegal” edges (i.e.,a node having more than one parent from the evidence

nodes)(Cohen, Goldszmidt, Kelly, Symons, & Chase,2004). Another procedure, sketched out in Algorithm1, builds a Minimum Weighted Spanning Tree (MWST)of the evidence nodes and then connects the fault modeto all of the evidence nodes in the tree (Friedman et al.,1997). We use this algorithm in our work. A standardalgorithm (e.g., Kruskal’s algorithm(Kruskal, 1956)) isapplied to generate the MWST. The edge weight compu-tation for the tree structure utilizes a log-likelihood crite-rion, such as the Bayesian likelihood value (Chickering,Heckerman, & Meek, 1997) and the Bayesian Informa-tion Criterion (BIC) (Schwarz, 1978). If the values arenaturally discrete or they represent discretized continu-ous values, the Bayesian likelihood metric is preferred.This is a simple metric, which calculates the likelihoodthat two variables are dependent. The BIC is bettersuited for data sets whose features are derived from con-tinuous distributions (like a Gaussian Normal). For ei-ther measure, the values are calculated for every pair ofevidence nodes and stored in a matrix. Note that thevalue calculated for node i to node j is different for thevalue calculated for node j to node i, Therefore, the di-rected edges of the MWST represent the implied direc-tion of causality, and the derived structure includes pre-ferred causal directions (and not just correlational infor-mation).

Figure 3: Example TAN Structure

An example TAN structure is illustrated in Figure 3.The root node, labeled class, is the fault hypothesis of in-terest. The other nodes represent evidence supporting theparticular fault hypotheses. For the structure in Figure 3,rolltime, a monitor associated with the shutdown phaseof the aircraft is the anchor evidence node in the TANstructure, called the observation root node. Like a Naı̈veBayesian classifier, the fault hypothesis node (class) islinked to all of the relevant monitor nodes that supportthis hypothesis. Dependencies among some of the mon-itors, e.g., rolltime and dipEGTC, are captured as addi-tional links in the Bayesian network. Note that the TANrepresents a static structure; it does not explicitly cap-ture temporal relations among the evidence. The choiceof the observation root node is important; in some ways,it represents an important monitor for the fault hypothe-

7


sis, since it is directly linked to this node. This means thedistribution used in the observation root node(whether itbe a discrete CPT, or a continuous distribution) is con-ditioned only on the priors of the class distribution. Therest of the MWST structure is also linked to this node.All other conditional probability tables(CPTs) generatedfor this TAN structure include the class node and at mostone other evidence node. The choice of the observationroot node may determine the overall structure of the tree,but for the same data, the procedure for identifying thestronger causal links should not change in any signifi-cant way, i.e., the strong links will appear on all TANs,irrespective of the choice of the observation root node.

Algorithm 1 TAN Algorithm Using MWST1: INPUT:Dataset D of N Features and a label C2: INPUT:Observational Root Node FRoot3: INPUT:CorrelationFunction4: OUTPUT:TAN Structure with Adjacency Matrix,

CLassAdjMat, describing the Structure5: OUTPUT:Probability Values ProbVec for each

Node {Note: Corr is a matrix of the likelihoodthat feature i is causally related to feature j (dif-ferent values can be found for i to j and j to i)}{Count(Node,ClassAdjMat,D) is a counting func-tion, that takes the Data, the Class, the Full Adja-cency Matrix of the TAN and for the Node findseither the CPT for discrete-valued features, or theset of means and covariances to describe the Gaus-sian Normal Distributions of the Node for continu-ous valued variables.} {AdjMat describes the par-ents so that correct data slices can be isolated andused in the counting. }

6: for featurei = 0 to featurei = N do7: for featurej = 0 to featurei = N do8: if featurei 6= featurej then9: Corr (i,j) = CorrelationFunction(fi, fj ,D)

10: end if11: end for12: end for13: AdjMat = MWST(Corr, FRoot){ Build a Minimum

Weighted Spanning Tree using the Correlation Ma-trix and the Root chosen}

14: for featurei = 0 to featurei = N do15: ClassAdjMat(featurei, C) = 1 {Connect ev-

ery feature to the Class Node to build the TAN}16: end for17: ProbVec(C) = Count(C,ClassAdjMat,D) {Estimate

the parameters, starting with the class}18: for featurei = 0 to featurei = N do19: ProbV ec(featurei) =

Count(featurei, ClassAdjMat, D)20: end for21: RETURN: (AdjMat, ProbV ec)

When the inference is used to assign a faulty or nomi-nal label to the observed flight data, the result will be bi-ased towards one class(fault) over another based on theCI value of the observation root node. This shift alsochanges some causal relationships and may impact howthe counting algorithm for parameter estimation groupsthe data and produces probabilities for the evidence. Ina later section we discuss how these choices can be usedby the domain expert to make effective improvements to

the reference model for the AHM.This choice of the observation root node, as shown in

Algorithm 1 is an input parameter to the algorithm. Thischoice is normally based on a ranking computed using aheuristic, such as the highest BIC value. The algorithmin Weka (Hall et al., 2009) builds TANs with every fea-ture as the root node of the MWST. It compares the gen-erated structures, using a scoring metric such as the log-likelihood for the training data. The structure with thebest score is then chosen as the classifier structure. An-other approach could use domain knowledge to choosethis node. For example, using expert knowledge of thesystem one may choose the sensor that is closest to thefault under consideration because it is not likely to becausally dependent on other sensors. The implication inthe classifier is that it will be closest to indicating a fault.

Consider the example TAN shown in Figure 4. Whenthe data for constructing the TAN is extracted fromflights just before the adverse event occurred, the rootnode chosen by the Weka scheme is idlespeed. Thisnode connects to the rest of the MWST, which in thiscase is the starttime feature, to which the rest of the fea-ture nodes are connected. Using data from flights thatwere further away (before) from adverse event occur-rence, the Weka algorithm picked PeakEGTC as the rootnode. This is illustrated in TAN structure in Figure 5.However, the derived causal link from idlespeed to start-time to a large group of nodes is retained at the bottomright of Figure 5. The similarities and shifts in the TANstructures from different segments of data typically in-forms the domain expert about the underlying phenom-ena due to the fault that is captured by the monitors. Wediscuss this in greater detail when we develop our casestudy in Section 7..

6.1 Implementations Used for Building TANsTwo different implementations can be employed for theTAN algorithms used in the experiments. The first isone that attempts to maintain the continuous nature ofthe features and build Gaussian Normal distributions forthe nodes. It is implemented in MATLAB using theBayesian Network Toolkit (Murphy, 2011).

The second method from the Weka (Hall et al., 2009)toolkit, uses a discretization algorithm which looks to bineach of the features into sets that unbalance the classesto provide the best possible split. For this case study, itproduced more accurate classifiers, however, there weresituations where it created a number of very fine splitsin the feature values to define all of the class structures.The result was excessive binning, which produced verylarge conditional probability tables. When considering amore general view, methods that produce excessive bin-ning are likely to be less robust to noise. Therefore, onehas to consider these trade offs when choosing betweenthese approaches.

7. EXPERIMENTSTo evaluate the data mining approach and demonstrateits ability to improve a diagnoser reference model, weconducted a case study using real flight data from theregional airline described earlier. We defined three stan-dard metrics: (1) classification accuracy (2) false pos-itive rate, and (3) false negative rate to systematicallyevaluate the TAN learning algorithm. Starting from the

8


Figure 4: TAN Structure with idlespeed as observation root node

Figure 5: TAN Structure with peakEGTC as observational root node

9


flight in which the adverse event occurred for a partic-ular aircraft, we used the earliest time to detection asanother metric to evaluate the improvement in the sys-tem reference model after it has been updated with newinformation generated from the TAN classifier structure.The particular case study discussed here was an aircraft,where an overheated engine caused the engine to shut-down. This was considered to be a serious adverse event,and the pilots decided to return to the originating airport.By comparing the data from the faulty engine against thethree other engines on the aircraft, which operated nor-mally, starting from 50 flights before the adverse event,we were able to generate additional monitor informationthat reliably pointed to a FuelHMA problem (i.e., leakat the fuel meter) 40 flights before the actual incident.We break up the case study into three experiments anddiscuss their details next.

7.1 Experiment 1The first task was to investigate the effectiveness of thegenerated classifier structures in isolating the fault con-dition using the condition indicator information derivedfrom the flight data. We used condition indicators (CIs)rather than the health indicators (HIs) in this analysisbecause they make fewer assumptions about the natureof the data. We hypothesized that the original expert-supplied thresholds for the HIs were set at conservativevalues to minimize the chances for false alarms, andour derived classifier structures could potentially pro-vide better thresholds without sacrificing the accuracyand false alarm metrics. This would lead to faster de-tection times.

From the ASIAS database, we extracted the aircraftand flight number in which the adverse event occurred.The report also indicated the nature of the fault thatcaused the adverse event, and knowledge of the fault pro-vided context to our domain expert as to when this faultcould be detected in the engine system. Our expert sur-mised that the fault would most likely start manifestingabout 50 flights prior to the adverse event. The initialdataset that we then formulated consisted of the calcu-lated CIs for all 50 of the identified flights. Each enginehas its set of monitors, therefore, we had four sets ofCIs, one for each engine. For analysis, we consideredtwo ways for organizing this data. The first approachcombined the four engine CI’s as four separate featuresassociated with one data point (i.e., one flight). Since wehad 25 different CIs, this meant the dataset consisted of50 data points, with each data point defined by 100 fea-tures. The second approach looked at each engine as aseparate data point. Therefore, we formed four datasets,each with with 50 data points and 25 features. From theproblem definition, it was clear that one of the four en-gines was faulty, and other three were most likely nomi-nal for the 50 flights that we were analyzing. Therefore,we chose the latter approach for representing the data. Tolabel the dataset appropriately for the classification studythat we applied in this experiment, the three engines ofthe aircraft that showed no abnormalities (1, 2, and 4)were labeled as nominal, and the data points correspond-ing to engine 3, where the shutdown incident occurred,was labeled as faulty.

The classifiers were trained and evaluated using 10-Fold Cross validation (180 samples for training, and 20for testing) with the nominal engine data being agnos-

tic of which engine produced the data. All of the CIsdescribed in 4. were used as features in the classifier al-gorithm. The TAN generation algorithm from the Wekatoolkit were used to derive the necessary classifiers. Thefact that the discretized representation of the conditionalprobabilities were employed made it easier to find thethreshold values for the diagnostic monitors that werelinked to each CI. This is discussed in greater detail inSection 7.3. The derived TAN structure is illustrated inFigure 6.

The classification accuracy for this TAN structure washigh, the average accuracy value was 99.5% with a .7%false positive rate and 0% false negative rate. These ini-tial results were encouraging and to better understandthem, the experiment was extended to confirm that theclassifier results were be attributed to the evolving faultin engine 3 and it was not just an artifact of the differ-ences between the different engine characteristics. Theabove experiment was repeated with the training dataincluding one of the nominal engines (1, 2, or 4) andthe faulty engine, 3. The other two nominal engineswere used as the test data. If the classifier split the re-maining nominal engine data between the nominal andfaulty classes derived, this would indicate that its struc-ture more likely an artifact of engine placement on theaircraft. This experiment was repeated two more times,each time using a different nominal engine providing thetraining data and the other two being used as the test data.For all 3 experiments, the fault classification accuracy re-mained high, indicating that the classifier was truly dif-ferentiating between the fault and no-fault conditions.

7.2 Experiment 2The positive results from the classification task led to thenext step, where we worked with a domain expert to de-termine which of the CIs in the classifier provided thebest discrimination between the faulty and nominal con-ditions. This information would provide the necessarypointers to update the current reference model. As a firststep, the expert examined the TAN created using datafrom the 50 flight set used in Experiment 1. The ex-pert’s attention was drawn to the complex relationshipbetween certain pairs of CI’s during different phases ofthe flight:(1) rolltime and dipEGTC during the Shutdownphase, and (2) PeakEGTC and Starttime from the Startupphase. The expert concluded that there was a likely de-pendence between the shutdown phase of flight n andthe startup of the next flight, n + 1. The reasoning wasthat an incomplete or inefficient shutdown in the previ-ous flight created situations where the startup phase ofthe next flight was affected. The expert hypothesized thatthis cycle of degradation from previous shutdown to thenext startup resulted in the fault effect becoming largerand larger, and eventually it would impact a number ofCIs of the faulty engine.

This phenomena was investigated further by designingan experiment to track how the causal structure and ac-curacy of the classifiers derived from different segmentsof data. The different segments were chosen as intervalsof flights before the flight with the adverse event occur-rence as shown in Table 1. The 50 flights were dividedinto 5 bins of 10 flights each. A test set was constructedfrom the remaining 40 flights (data with nominal andfaulty labels) as well as the samples of CIs from engine3 after it had been repaired (after engine 3 was repaired,

10


Figure 6: TAN Structure Generated using Data from all 50 Flights

Bin Training Flights Acc.on Holdout Set FP% Obs. Root Node Children of ORN Notes1 1 to 10 97.65% 2.30% IdleSpeed StartTime Thresholds Chosen

from this Bin due tolow FP

2 11 to 20 93.90% 5.70% peakEGTC liteOff,dipEGTC peakEGTC Impor-tant Node

3 21 to 30 94.65% 5.30% peakEGTC liteOff,dipEGTC peakEGTC Impor-tant Node

4 31 to 40 96.62% 3.50% startTime peakEGTC Links startTime andPeakEGTC

5 41 to 50 96.06% 4.10% liteOff phaseTwo,RollTime Links Startup andRolldown CI

Table 1: Accuracy, False Positive Rate, Observational Root Node and Immediate Child Node for Classifiers Createdfrom different data segments

11


no anomaly reports were generated by mechanics, andno further adverse events were reported for engine 3 inthe ASIAS database, therefore, it was considered to benominal). Table 1 shows the accuracy and false posi-tive rate(FP%) metrics reported for the five experiments.The observation root node, and its immediate child in thegenerated TAN structures are also shown.

The conventional wisdom was that the accuracy andfalse positive metrics would have the best values for theclassifiers generated from data close to the adverse event,and performance would deteriorate for the TAN struc-tures derived from bins that were further away from theincident. The actual results show partial agreement. Thebin 1 experiment produced the highest accuracy and low-est false positive rate, but the next best result is producedfor TAN classifiers generated from the bin 4 data. Thisprompted the domain expert to study the bin 1 to bin 4TANs more closely. The expert concluded that two CIs,startTime and peakEGTC showed a strong causal con-nection for bin 4, and startTime was highly ranked forthe bin 1 TAN. On the other hand, PeakEGTC was theroot node for bins 2 and 3. This study led the domainexpert to believe that a new monitor that combined start-Time and peakEGTC would produce a reference modelwith better detection and isolation capabilities. The pro-cess of designing and testing the diagnoser with the newmonitor is described as Experiment 3.

7.3 Experiment 3Working closely with the data mining researchers, thedomain expert used the framework in the problem state-ment to reconcile the results from Experiment 2 to sug-gest explicit changes in the reference model; this in-cluded: (1) updates to the threshold values that specifiedthe diagnostic monitors (i.e., updated HIs from the CIs),(2) a new monitor that could be added to the current ref-erence model and (3) the addition of a “Super Monitor”to the reference model.

The CPTs generated by the learned classifier were de-fined as discretized bins with split points that could beinterpreted as thresholds. Looking at the bins, the lowestfalse positive rate occurred in bin 1. For the observationroot node, the thresholds were updated using the resultsfor bin 1 by comparing the split values with the origi-nal thresholds. For the remaining nodes, their causalitywith respect to the observation parent was removed bymarginalizing to remove that particular variable. Oncemarginalization is applied, the CPT lists the probabil-ity values at the nominal versus faulty split points. Thedomain expert studied these split values and made de-cisions on whether the new split values should updatethe thresholds in the reference model. The trade off wasto improve the accuracy of fault detection without in-troducing too much noise (uncertainty) into the decisionprocess.

Studying the TAN structures provided additional in-formation to the domain expert. When the slowStart HIfired, the expert discovered that this was not because thestartTime during start up was slow; sometimes the faultoccurred when the startTime was too fast. This implieda new HI could be added to the failure mode that nowexamines if startTime is under a threshold and too fast.The addition of this HI(called fastStart) to the referencemodel would be to speed up detection by adding newevidence to indict the fault.

Experiment 2 also showed a causal relationship ap-pearing between startTime and peakEGTC. The domainexpert suggested adding this as a “super monitor”. Thisnew HI would combine information from the fastStartHI and the HighTemp HI to identify the fuelHMA faultin the reference model. In other words, if both moni-tors fired, then this new monitor would also fire directlyimplicating the fault hypothesis. In other words, jointoccurrence of these two monitors provides stronger evi-dence of the fault than if one considers the effect of thetwo monitors individually. For example, in the origi-nal structure that showed a possible relationship betweenmonitors in flight N and flight N+1, the causality mightcause this new monitor to fire only when the two HI in-volved fire in that explicit sequence, flight n and flightn + 1. Not only does this super monitor combine theresults from other monitors, but it also indicates cyclicbehaviors that again provide very useful diagnostic infor-mation. In general, these “super monitors” could modelcomplex interactions thus increasing the overall discrim-inability properties of the reasoner. The consequence ofusing a super monitor, is that the usefulness of the twomonitors used in the construction are lost. These are re-moved from the links to the failure mode being exam-ines(however they remain for any other failure mode).In this situation, the just created monitor for fast start-Times would be removed as well as the HighTemp HI inplace of a new super monitor for a fast start and a highengine temperature on start up.

To show that this new super monitor and the updatedthresholds produce better results, multiple traces of themonitor output were made for the 50 flight data set. Thisincluded 10 nominal flights after the problem was caughtand corrected. The first trace is only a recording of theoriginal monitors designed by the expert. The secondtrace includes the new monitors(both the fast startTimemonitor and “super monitor”) derived by the data mininganalyses, as well the updated information(thresholds).Run separately, they can be analyzed to determine if thereasoner finds the fault sooner in the trace and indicatesthat maintenance is more than likely needed for the air-craft.

These results from the reasoner simulations are shownin Figures 7 and 8. The traces illustrate the reasoner’sinferences at different flight numbers before the actualincident occurrence. This analysis demonstrates how farbefore the adverse event the reasoner would reliably de-tect the fault, and potentially generate a report that wouldlead to preventive maintenance, and, therefore, avoid-ance of the adverse event. With the original referencemodel the reasoner was unable to disambiguate betweenthree potential fault candidates at any point leading upto the event. All of the fault candidate hypotheses re-quired more evidence to support the isolation task. Thiswould not avoid the unfortunate shutdown and the emer-gency return to the originating airport. Figure 8 showsthe reasoner trace for the new reference model. Using theupdated thresholds and the new super monitor(which isderived from a new monitor itself and one original moni-tor) suggested by the data mining algorithms led to a cor-rect isolation of the fault, i.e, the fuelHMA problem. Inthis case, the reasoner originally hypothesized five faultconditions: four of these were linked to the faulty en-gine and one was a vehicle level hypothesis. As furthermonitor information became available, fuel metering re-

12


Figure 7: Trace of the Reasoner on the Original Reference Model

Figure 8: Trace of the Reasoner with the improved Reference Model

13


mained the only plausible candidate, and the fault hy-pothesis was established unambiguously. The fact thatthis isolation by the reasoner occurred 30 flights beforethe incident is significant because it gave sufficient ad-vanced warning to the maintenance crews to fix the prob-lem before an emergency situation developed.

This case study provides encouraging results in thearea of diagnoser model improvement through data min-ing. It indicates that it may be possible to uncover newinformation about the relationship between componentson a vehicle and how they can be harnessed to improvediagnostic reasoning. Not only can it help isolate faults,but also potentially catch them earlier in the cycle. Thesethree experiments provide a general direction to assist-ing a domain expert in improving their work, and givingthem access to new or missing information.

8. CONCLUSIONS AND FUTURE WORKThe overall results from this case study generated pos-itive results and show the promise of the data miningmethodology and the overall process that starts from datacuration and ends with systematic updates to and verifi-cation of the system reference model. The results pre-sented clearly demonstrate that the data mining approachis successful in: (1) discovering new causal relations inthe reference model, (2) updating monitor thresholds,and (3) discovering new monitors that provide additionaldiscriminatory evidence for fault detection and isola-tion. Experiment 3 demonstrated that the new knowl-edge leads to better diagnoser performance in terms: (1)early detection, and (2) better discriminability. An im-mediate next step will be to generalize this methodologyby applying it to other adverse event situations. In thelonger term, to further validate this work, we plan to ad-vance this research in a number of different directions.• Validation of the approach and classifier structures

generated by looking at additional engine data setsfrom other flight data that report the same and re-lated adverse events. To establish the robustness ofthe work, it is important to extend the analysis tolooking at multiple occurrences of the same adverseevent, and to compare the thresholds, relations, andmonitor structures generated by the extended dataanalysis.

• Extension of the analysis methodology beyond sin-gle systems and subsystems. A rich source of infor-mation about fault effects involves looking at theinteractions between subsystems, especially afterfault occurrence begins to manifest. Of particularinterest is looking at cascades of monitors and cas-cades of faults. In this framework, studying the re-sponse of the avionics systems under different faultconditions would be very useful.

• Advance our data mining techniques to extractcausal relations between avionics and other sub-systems, as well as study correlations between thecombined avionics and engine features and adversevehicle events, such as in-flight engine shutdownsand bird strikes. Understanding what features ofa flight differentiate situations when a pilot startscompensating for what may be a slowly degrad-ing component that originates from a bird strikewill help us gain a better understanding of how to

monitor the actual operation of the aircraft withits subsystems under various conditions. This alsopresents interesting problems from the applicationand development of machine learning algorithms toutilize in this data mining problem.

ACKNOWLEDGMENTSThe Honeywell and Vanderbilt researchers were partiallysupported by the National Aeronautics and Space Ad-ministration under contract NNL09AA08B. We wouldlike to acknowledge the support from Eric Cooper fromNASA; Joel Bock and Onder Uluyol at Honeywell forhelp with parsing and decoding the aircraft raw data.

REFERENCESCheng, J., Greiner, R., Kelly, J., Bell, D., & Liu, W.

(2002). Learning Bayesian networks from data:An information-theory based approach. ArtificialIntelligence, 137(1-2), 43 - 90.

Chickering, D. M., Heckerman, D., & Meek, C. (1997).A Bayesian approach to learning Bayesian net-works with local structure. In In Proceedings ofThirteenth Conference on Uncertainty in ArtificialIntelligence. Morgan Kaufmann.

Cohen, I., Goldszmidt, M., Kelly, T., Symons, J., &Chase, J. S. (2004). Correlating instrumentationdata to system states: a building block for auto-mated diagnosis and control. In Proceedings ofthe 6th conference on Symposium on OpeartingSystems Design & Implementation - Volume 6 (pp.16–16). Berkeley, CA, USA: USENIX Associa-tion.

Dearden, R., & Clancy, D. (2001). Particle filters forreal-time fault detection in planetary rovers. InProc. of the 12th International Workshop on Prin-ciples of Diagnosis (p. 1-6).

Friedman, N., Geiger, D., & Goldszmidt, M. (1997).Bayesian Network Classifiers. Machine Learning,29, 131–163.

Grossman, D., & Domingos, P. (2004). LearningBayesian network classifiers by maximizing con-ditional likelihood. In Proceedings of the twenty-first international conference on Machine learning(pp. 46–). New York, NY, USA: ACM.

Hall, M., Eibe, F., Holmes, G., Pfahringer, B., Reute-mann, P., & Witten, I. H. (2009). The WEKAData Mining Software: An Update. SIGKDD Ex-plorations, 11(1), pp. 10-18.

Kruskal, J., Joseph B. (1956). On the Shortest Span-ning Subtree of a Graph and the Traveling Sales-man Problem. Proceedings of the American Math-ematical Society, 7(1), pp. 48-50.

Lerner, U., Parr, R., Koller, D., & Biswas, G. (2000).Bayesian Fault Detection and Diagnosis in Dy-namic Systems. In Proc. of the 17th Nat. Conf.on Artificial Intelligence (p. 531-537). San Mateo,CA, USA: AAAI Press.

Murphy, K. (2011). Bayesian Net Tool-Box @ONLINE. Available fromhttp://code.google.com/p/bnt/

Pearl, J. (1988). Probabilistic reasoning in intelligentsystems: networks of plausible inference. MorganKaufmann Publishers Inc.

14


Roychoudhury, I., Biswas, G., & Koutsoukos, X. (2008).Comprehensive diagnosis of continuous systemsusing dynamic Bayes nets. In Proc. of the 19thInternational Workshop on Principles of Diagno-sis (p. 151-158).

Schwarz, G. (1978). Estimating the Dimension of aModel,. Annals of Statistics, 6.

Smyth, P. (1994). Hidden Markov models for fault de-tection in dynamic systems. Pattern Recognition,27(1), pp. 149-164.

Spitzer, C. (2007). Honeywell Primus Epic Aircraft Di-agnostic and Maintenance System. Digital Avion-ics Handbook(2), pp. 22-23.

Verma, V., Gordon, G., Simmons, R., & Thrun, S.(2004). Real-time fault diagnosis. IEEE Roboticsand Automation Magazine, 11(2), pp. 56-66.

15

deriving bayesian classiﬁers from flight data to enhance ... · a reference model that provides a...

Documents