1

Project Echo and Speech

Recognition Software

Matthew N. GrayNIFS Student Intern – Summer

2016Airspace Operations and

Safety ProgramCrew Systems and Aviation

Operations Branch Mentors: Dr. Angela Harrivel,

Chad Stephens

2

Crew State Monitoring (CSM) – Motivation

• National airspace is becoming increasingly busy and more complex.

• If we could stop just one fatal commercial aviation accident, we could save hundreds of lives and a billion dollars.

• Loss of Control – Inflight (LOC-I)o Largest Category of aircraft-related Fatal Events

• CSM aims to improve pilot operational efficiency during safety-critical operations by: o Improving the human-machine interface in

aircrafto Aiding in pilot attention training 3 Boeing Statistical Summary of Commercial Worldwide Jet

Transport Accidents, 2011. Includes only accidents involving turbofan or turbojet airplanes with max takeoff weight > 60,000 lbs., referenced in the CAST Airplane State Awareness Joint Safety Analysis Team Final Report, June 17, 2014

3

Background – Crew State Monitoring (CSM)

• Data Collectiono fNIRS, EEG, EKG, Resp., GSR, Eye-

tracking, etc.

• Data Synchronizationo MAPPS Software

• Signal Pre-Processingo Filtering, Feature Extraction

• Machine Learningo Classification Algorithmo Model Evaluation

• Real-time State Indicator Displayo High/Low Workload, Distraction,

Inattentional Blindness, Confirmation Bias, etc. Figure courtesy of Charles Liles

and the LaRC Big Data and Machine Information Team

4

Background – AFDC 2.0 and SHARP 1.0

• Augmented Flight Deck Countermeasures (AFDC 2.0)o Seeks to improve human-

machine interaction among safety critical operations of airplanes

• Scenarios for Human Attention Recovery using Psychophysiology (SHARP 1.0)o Supports CAST goals by

measuring crew cognitive states in simulators during safety critical operations SHARP Display

Muse

SmartEye

Spire

Empatica E4

BIOPAC fNIR100B

EEG

fNIRS

Heart Rate (PPG)GSR

Temp.Accel.

Respiratory Rate

Eye-tracking

Project Echo

Sensors

Data Acquisitio

n

Data Synchronizat

ion

MAPPS or MAPPS Equivalent:• Lab Streaming Layer (LSL)• iMotions• XTObserver

Signal Processin

g

Classification

Algorithm

Hidden Markov Models (HMMs)

Neural Netowrks (NNs)

Machine Learning Display

Indicate Cognitive State:• High/Nominal Workload• Channelized Attention• Diverted Attention• Confirmation Bias• Inattentional Blindness

6

Project Echo – SmartEye® Eye-Tracking System

• Hardware Setup• Camera Calibration• Building World Model• Head Profile Creation• Gaze Calibration• Real-time data transfer

(future)

SmartEye® Setup with corresponding World Model

7

Project Echo – Empatica E4 Wristband

• Measures oHeart Rate – HR (PPG)oGalvanic Skin Response – GSRo Skin Temperatureo Acceleration/Movement

• Lightweight, non-invasive, portable

• Real-time data streaming through Matlabo TCP/IP Connectiono Stored in text file

8

Speech Recognition Software – Motivation

• Talking could affect classification accuracy during runsoMore/varying cognitive

activationo Irregular breathingoMovement

• Automatic labeling of speech vs. other noises for future analysis

Talking

9

Speech Recognition – MFCCs

• Mel Frequency Cepstral Coefficients (MFCCs)oFeature derived from audio signal oRepresents shape of vocal tract

through short-time frequency analysis

oApplies filter to frequency response of signal to emulate human hearing

Humans can distinguish low frequencies easier than high frequencies

Cross-sectional shapes of vocal tracts

Cochlear frequency map showing logarithmic frequency resolution of human hearing

10

Speech Recognition – MFCCs (cont.)

Steps:• Calculate moving window PSD

o Assume ‘stationarity’ at 25ms window sizeso Apply Hamming filter to windowed signal

• Create Mel filter bank (shown to the right)• Multiply each windowed PSD by each

filter in filter bank• Sum powers in each binned and filtered

frequency for each frame• Take log and discrete cosine transform of

summed powers• Produces array of 12 coefficients for

each windowed time series

×

=

11

Future Work

• Acquire data real-time from SmartEye® system

• Parse out data stream from Empatica in Matlab into separate text files

• Synchronize data streams from all devices with MAPPS or MAPPS equivalent (LSL, etc.)

• Incorporate in-house preprocessing and machine learning scripts

• Input MFCCs from audio signals into Hidden Markov Model machine learning classifier

12

Acknowledgements

• Angela Harrivel, PhD• Chad Stephens, PhD Candidate• Kyle Ellis, PhD• Ray Comstock, PhD• Kellie Kennedy, PhD Candidate• Nick Napoli• Katrina Colucci-Chang• Will Hollingsworth• Alex Liang