augmented cognition and the cognitive cockpit lt jefferson d. grubb msc usn navair human systems...
TRANSCRIPT
![Page 1: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/1.jpg)
Augmented Cognition and the Cognitive Cockpit
LT Jefferson D. Grubb MSC USNNAVAIR Human Systems Department
![Page 2: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/2.jpg)
Aircrew Information Needs
• Flight Information– Speed, altitude, attitude etc.
• Systems information– RPM, temperature, voltage, fuel etc.
• Navigation information– Where am I? Where is the airport? Where is that
mountain?
• Mission Information– Targets, Traffic, Threats, etc
![Page 3: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/3.jpg)
Evolution of the Navy Cockpit
![Page 4: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/4.jpg)
Effective Cockpit Design
• Cockpits are designed to a happy medium
![Page 5: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/5.jpg)
The Problem
• Cockpits are static
• Pilots’ information needs and capabilities are not– Performance changes with
arousal– Modalities matter
Central Executive
Pho
nological loop
Visuospatial
sketchpad
An adaptive cockpit would have to read the pilot’s mind
![Page 6: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/6.jpg)
Reading the Mind
• Cognitive state is brain state
• Neuroimaging techniques monitor brain state
![Page 7: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/7.jpg)
Neuroimaging Techniques
• Electromagnetic techniques– EEG/ERP, MEG/ERF
• Measure electromagnetic consequences of brain activity
– Timing of brain activity– Intensity of brain activity
• Hemodynamic techniques– fMRI, PET, fNIRS
• Measure changes in blood flow and oxygenation associated with brain activity
– Location of brain activity» Different locations are
associated with different functions
![Page 8: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/8.jpg)
AugCog and CogPit• DARPA’s Improving Warfighter
Information Intake Under Stress (formerly Augmented Cognition) Program– Apply neuroimaging to solve human
factors problems• Cognitive Cockpit (CogPit)
– Joint QinetiQ/Alion/NAVAIR project to apply neuroimaging to aviation
![Page 9: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/9.jpg)
Constraints on Techniques
• Equipment– Cost– Size– Power consumption– “Weird Stuff”– Comfort
![Page 10: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/10.jpg)
Constraints on Techniques
• Signal processing
![Page 11: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/11.jpg)
Techniques of Choice
• EEG– Spectral Power Density– Interelectrode coherence
• fNIRS– Blood Oxygenation
![Page 12: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/12.jpg)
How it Works
• Hook the “pilot” up
![Page 13: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/13.jpg)
Calibrate
• 3 levels of tracking task
• 3 levels of Bakan task
• 4 combinations of Bakan and Tracking
5
![Page 14: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/14.jpg)
CogMon• Records spectral power and
interelectrode coherence
• Produces linear model relating physiological signal to different levels of tracking and Bakan task
• Rates subsequent recordings against model to produce judgment of pilot spatial and verbal workload in real time
![Page 15: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/15.jpg)
ATAS Cockpit
• Low cost, maximally open source
• Allows great flexibility to prototype “mitigation strategies”
![Page 16: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/16.jpg)
Knowledge Coordinator
• Assesses pilot workload and mission context.
• Triggers workload mitigations
![Page 17: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/17.jpg)
CogPit Closed Loop
Control Inputs
Information
Mitigations
Mission Context
Cognitive Workload Measurement
EEG/fNIR
![Page 18: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/18.jpg)
• Attention– Declutter
– Alerts
• Executive Function– Function Automation
• Working Memory– Sequencing
• Sensory Input– Multimodal Cueing
Mitigations
![Page 19: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/19.jpg)
Does it work?
Tests at NAVAIR are just getting underway• Specifics
– Detection of cognitive state• Recent results from CogPit are promising• Actual flight tests of EEG workload monitor by
other labs (AFRL, NRC Canada/U. of Iowa)
– Mitigation Strategies• Too early to tell for current CogPit• Needs more focus
![Page 20: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/20.jpg)
Will the user accept it?
You make the call…
![Page 21: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/21.jpg)
Road Ahead
• The full closed-loop technology is not ready– Imaging equipment is bulky, temperamental,
and uncomfortable– Mitigations are still in development
• Bits of the technology could be useful in the near term.
![Page 22: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/22.jpg)
First Step: Hypoxia?
• Poses danger to all high altitude operations• Factor in at least 16 mishaps (3 fatalities) in Naval aviation since 2001
•Naval Aviation has lost 0 TACAIR assets to enemy fire since Desert Storm
![Page 23: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/23.jpg)
Hypoxia
•Currently mitigated with altitude chamber training
•Symptoms are subtle and variable
•Fundamentally strikes at cognitive ability
•Current countermeasure hinges on ability of broken system to diagnose itself
![Page 24: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/24.jpg)
Hypoxia Detection with fNIRS
• fNIRS passes near infrared light through the skull and measures absorption of different wavelengths– Oxygenated and
deoxygenated blood preferentially absorb different wavelengths
– Yields measure of both blood volume and blood oxygenation
![Page 25: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/25.jpg)
• fNIRS potentially provides the basis for a hypoxia and G-LOC warning system
• Open questions:– How reliable would such a system be?– What do we do with the information?
![Page 26: Augmented Cognition and the Cognitive Cockpit LT Jefferson D. Grubb MSC USN NAVAIR Human Systems Department](https://reader034.vdocument.in/reader034/viewer/2022051401/56649d1f5503460f949f2373/html5/thumbnails/26.jpg)
Take Away
• CogPit is a platform to develop cockpits that “read the pilot’s mind” to provide the pilot with the right information at the right time
• The full technology is still a ways off, but subsets of it may be able to provide benefits to aviators much sooner