Detection of Precursors to Aviation Safety Incidentsdue to Human Factors

I. Melnyk, P. Yadav, M. Steinbach, J. Srivastava, V. Kumar and A. Banerjee

Department of Computer Science & EngineeringUniversity of Minnesota, Minneapolis

ICDM Workshop on Domain Driven Data MiningDecember 7, 2013

2

• Introduction

• Related Work

• Proposed Approach• HMM vs HSMM

• Experimental Results

• Conclusion

Overview

3

• Fatal accidents and onboard fatalities 2003-2012 [Boeing Report ’12]

Introduction

4

• Estimated two-fold increase in air traffic by 2025 [Sheridan ’06]

• Congestion in air and airports• Load on pilots and traffic controllers• Greater chance to make error

• Our objective• Detect precursors to aviation safety incidents due to human factors

– Analysis and modeling of pilot actions– Data generation and evaluation

Introduction

5

• Hidden Markov Models [Srivastava ’05]• Observations modeled using N-dim binary vector (switches in cockpit)• Cluster data to get smaller class of observations; build HMM over reduced data

• Clustering approach [Budalakoti et al. ’09]• Cluster pilot action sequences using k-medoids based on nLCS• Rank order sequences to identify anomalous sequences

• One-class SVM [Das et al. ’10]• Detects anomalies in both continuous and discrete sequences• Employed Multiple Kernel learning: LCS for discrete, SAX for continuous

• Dynamic Bayesian Networks [Saada et al. ’12]

• Hidden nodes – pilot actions; Observable nodes – aircraft sensors• Detects pilot errors in the past given current instrument data

Related Work: Aviation Safety

6

• Given database of normal pilot action sequences

• Actions come from finite alphabet:• Examples: “raise landing gear”, “lower flaps”, “decrease throttle”, etc.• Construct model of a normal sequence from data

• Assign anomaly score to a test sequence

• Entire sequence is anomalous (offline anomaly detection)• Specific action is anomalous (online anomaly detection)• Examples: “unusual order of actions”, “forgotten action”, etc.

Problem Formulation

7

• Flight phases

• Example: landing phase pilot actions

Analysis of Pilot Actions

Action ID

Time

descent touch down braking on runway

stage duration

time btw actionsNull action

8

• Flight phases

• Example: landing phase pilot actions• Simplification: ignore time duration between actions

Analysis of Pilot Actions

Action ID

Time

descent touch down braking on runway

stage duration

9

• Hidden states• Stages of aircraft operation• Examples: initial descent, touch down, braking, etc.

• Observations• Pilot actions• Example: initial descent – reduce throttle, lower flaps, lower landing gear, etc.

• Model parameters• Prob. distributions: Transition , observation , prior

• Drawback• Geometric state-duration distribution – encourages fast state switching• Inability to model arbitrary state durations

Hidden Markov Model (HMM)

10

• Additional hidden variable• State duration• Forces hidden state to last time steps

• Model parameters• Probability distributions:

• Duration

• Transition

• Observation , initial distributions ,

Hidden Semi-Markov Model (HSMM)

11

• Estimate probability distributions (conditional probability tables)• Duration• Transition• Observation• Initial distributions

• Use database of normal pilot action sequences

• Select parameters which maximize likelihood of data

• Non-convex problem without closed-form solution• Use Expectation Maximization (EM) [Dempster et al. ’77]

• Similar to Baum-Welch algorithm for HMM [Baum et al. ’70]

HSMM: Model Parameters Estimation

12

• Detect if a test sequence is anomalous

• Entire sequence is anomalous (offline anomaly detection)• Normalized joint log likelihood

• Specific action is anomalous (online anomaly detection) • Conditional probability

• Computational complexity • Computation uses Junction Tree algorithm for inference

• Cost:• For comparison, complexity for HMM:

Anomaly Detection Methodology

- sequence length- number of hidden states- maximum state duration

13

• Compared HMM and HSMM to detect duration anomalies• Data:

Results: Synthetic Data

Normal Anomalous

Training 200 0Testing 25 25

14

Flight Simulator

• FlightGear flight simulator

• Simulator setup• Landing flight phase• Cessna 172 Skyhawk landing at Half Moon Bay, CA airport• Aircraft controlled using keyboard

• Keystrokes interpreted as pilot actions• 12 commands to control aircraft

15

Pilot Actions for Landing

16

Flight Simulator Data

• Generated Data

• Types of anomalies• 1 – Throttle kept constant, flaps are not lowered; rest is normal• 2 – No initial throttle increase; rest is normal• 3 – Flaps are not lowered; rest is normal• 4 – At the end of flight brakes are not applied; rest is normal• 5 – Pilot overshoots runway and lands behind it; rest is normal

Normal Anomalous

Type 1 2 3 4 5

# of sequences

110 10 10 10 10 10

17

• AUC based on 11-fold cross-validation

• 110 normal sequences split into 11 parts• 10 parts used for training• 1 part + 50 anomalous sequences used for testing• Initialization was fixed across runs• Performance metric: area under ROC curve (AUC)

AUC Results: HSMM vs HMM

18

• Dependency of AUC on initialization

• Selected 10 random model initializations• Split of dataset was fixed across runs

• Training: 100 normal sequences• Testing: 10 normal + 50 anomalous sequences

• Performance metric: area under ROC curve (AUC)

Effect of Initialization: HSMM vs HMM

19

• Detected anomalies in streaming data1: No initial throttle increase2: Incorrect usage of rudder3: Mistakenly used elevator control after touch down

Offline vs Online Anomaly Detection

Anomaly scoring: Anomaly scoring:

20

• Baseline methods [Chandola et al. ’08]

• EFSA: Extended Finite State Automata• t-STIDE: Threshold-based sequence time delay embedding• WIND: Window-based anomaly detection

• Setup• Training: 100 normal sequences; Testing: 60 sequences• Sliding window length (history length) was varied from 1 to 20• Selected model with best AUC on training set; Evaluated on test set

HSMM vs Other Methods

Type 1 Type 2 Type 3 Type 4 Type 5

HSMM 1.00 0.97 0.77 0.88 1.00HMM 0.87 0.60 0.71 0.95 0.99WIND 0.9 0.60 0.70 0.87 1.00EFSA 0.85 0.67 0.67 0.91 0.98

t-STIDE 0.84 0.67 0.68 0.92 1.00

21

• Proposed framework to model discrete pilot actions• HMM

• Hidden states – stages of aircraft operation• Observations – pilot actions• Drawback – inability to model arbitrary state durations

• HSMM• Introduces additional hidden variable to model state durations

• Evaluated model performance• Synthetic data• Flight simulator data• Compared HSMM to HMM and other anomaly detection algorithms

Summary

Thank you!