Brain-based Technologies for Design

and Evaluation of Complex Systems

Trish Ververs, Engineering Fellow

October 2013

Honeywell.com

2

Overview

• Neurotechnologies

• Why brain-based systems

• Closed loop cognitive systems

• An example of use in design and evaluation

Honeywell.com

3

What is Neurotechnology

• Neurotechnology is a field of science that studies the body and

mind through the nervous system by electronics and mechanisms

• Neuroergonomics is the application of neuroscience to ergonomics

through the understanding biological explanations

Honeywell.com

4

Chaotic Situations

Air France

A330 Flight 447

Accident,

June, 2009

US Airways

A320 Flight 1549

Accident,

January, 2009

Honeywell.com

5

Human Computer Interaction

PROBLEM

• Increased system complexity

• Lack of transparency

• System provides visual, aural, tactile information

via displays, alerts, graphics on system state

SOLUTION

• Leverage human capabilities

• Make a system easy to learn and easy to use

• Provide a real-time assessment of user’s cognitive effort using the

system

• Provide a direct and natural means of interaction with the system;

not limited to mouse clicks and buttons presses

Note: Large pipeline from system (A), small pipeline to system (C)

Honeywell.com

6

Real Time Cognitive Assessment Technologies

Eyetracking/Pupillometry Neuro-imaging (fMRI)

EEG assessment

Physiological measures

Speech

Honeywell.com

7

Examples of Measures

Transcranial Doppler

temporal resolution

spatial resolution

direct measure of brain activity

transferable to real world applications

fMRI

EEG Spectral Power

Honeywell.com

8

Generic Neuro-based System Overview

Behavioral Protocol

System State

Video/Audio

System Inputs

…

Sensing

EEG ECG fNIR

Signal Processing

Artifact Detection

Artifact Correct. Spectral Transf.

Estimation

Working Memory

Drowsiness

Attention

Info Proc Speed

Cognitive State Assessment

Performance

Task Executive Cognitive State

during Work

Intelligent

Assistance

Context

User Models

Situation Models

Honeywell.com

9

Augmented Cognition Test Event

• Aberdeen Test Center, Mulberry Point

• Two-week, full platoon MOUT training – North Carolina National Guard 252nd Combined Arms

Battalion – Force on force exercises with simunitions – Multiple task performance under time pressure and

fatigue

• 24-hour exercise (7 hr blocks) – Exercises: Route Reconnaissance, Cordon and Search,

Long Hasty Defense – Mission: Low & high workload events

• Purpose: – Assess cognitive workload in real-time of team leaders

based on task engagement – Test ruggedness of sensors

• Mission Goals: Improve survivability and

increase mission effectiveness

Aberdeen Test

Center

USARIEM

Human

Research

Engineering

Directorate

Development Test

Command

Battle Lab

Integration Team,

Natick Soldier

Systems

NC Army National

Guard

Real-time Cognitive State Assessment of the Dismounted Warfighter

A

T1

T2

B

C

D E

FG

HI

J

Honeywell.com

10

Characterization of Workload

• Tested classifiers on other high and low periods to evaluate our ability

to classify mental workload during operational exercises

• Focus: Platoon Leader (PL) and Platoon Sergeant (PSG)

– Results fused information from EEG spectral power, ECG spectral power,

and ECG inter-beat interval (IBI)

– Presented for 2 classification sampling techniques (red= 2 fold, green= 10

fold) across different temporal smoothing windows (seconds on X axis)

base 2.5 5 10 20 40 60 0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1classification as a function of temporal smoothing

smoothing window (seconds)

Az

2-fold x-val

10-fold x-val

Platoon Leader Platoon Sergeant

Classification performance on Y axis (note: Az = .50 chance)

Honeywell.com

11

The Future of Neurotechnologies for Aviation

Where do we go from here?

• Electronic pilot assistant

• Single pilot operations – Reduce General Aviation accident rate

• Transoceanic flights – Reduce crew requirements, monitor

fatigue

• Unmanned aerial vehicles – remotely piloted vehicles

Honeywell.com

12

Where Will We See Neurotechnology…

• Specialized Training

• Military and Defense Systems

– Intelligence Analysts

– Unmanned Aerial/Ground Systems

– Cognitive Rehabilitation

– Readiness for Duty

• Automotive Systems

• Gaming

• TSA Monitoring

• Medical or Behavioral Treatments

– Alzheimer’s

– Spinal cord injury

– Traumatic brain injury

– Addiction

– ADHD