flight data handling with augmented reality · flight data handling with augmented reality doctoral...

Flight Data Handling with Augmented Reality

Doctoral Symposium

ICRAT ´18, Castelldefels, Barcelona (Catalonia)

June 25th – 29th 2018

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29 DLR.de/fl • Chart 1

Hejar Gürlük DLR Institute of Flight Guidance Controller Assistance

www.DLR.de/fl • Chart 2 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Background

Scope of Thesis

Concept

Summary & Outlook

Safety

Highest incidents and accident rates are related to “Runway” (62%) in contrast to “in flight” (3%) (ICAO Safety Report 2015)

Tower: Errors relate mainly to the field of visual perception major contributing factor for runway incursions (Hilburn, 2004)

Perception Memory DecisionMaking

ResponseExecution

Rule Breaking

Error types related to tower environment

Loss of Control in Flight

Runway

Accidents per flight phase

Accidents Fatalities

www.DLR.de/fl • Chart 4

„Head Down vs. Head Up Problem“

Head Down (HD): • Information systems and devices; frequent

usage but scan times not as long as head up (Pinska, 2009)

Head Up (HU): • Outside view

• Crucial for maintaining adequate situational

awareness (Hilburn, 2004)

• Remains most important source of information (Ruffner, 2008)

Tower Controller Working Position

Head Up

Problem: frequent transitions from HU to HD: “…delayed detection and interpretation of visual information…major error source for runway incursions“ (Hilburn, 2004)

Head Down

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Information Complexity

• Widely distributed head-down displays

• Controllers have to mentally merge the different information sources (HD / HU)

• Increasing information complexity (System Wide Information Management)

This can lead to higher workload in terms of

prolonged times for information acquisition and analysis

Tower Controller Working Position

Low Visibility Conditions

decreased capacity (throughput) and increased delay at airports operating under CAT II/IIIA-C

Background

Scope of Thesis

Concept

Summary & Outlook

Solution Approach and Aim of the Thesis

• Projection of relevant flight and topological data into the real outside

view by means of augmented reality

• prolongation of head-up times,

• improved situational awareness,

• reduced information search

• maintain capacity (even under low visibility conditions)

safety benefits

• Context-adaptive information presentation: user and task-dependent

presentation of only operationally relevant information

• What to present? When to present? How to present?

• reduction of controller workload (information acquisition and -analysis)

Motto and Aim: display the right (amount of) information at the right time!

Augmented Reality

Issues: - overclutter - untailored information presentation - missing interaction or feedback - integration of AR into operations unclear - display ergonomics, perceptual issues, registration errors

Research gap: • operational concept for such an assistance

system (e.g. for different operational conditions) -> HOW TO WORK WITH THIS?

• systematic assessment of operational and human performance benefits

overclutter

Main Research Question of Phd Thesis

What are the human performance and operational

benefits of context-adaptive augmented reality

for air traffic control towers?

Scope of Phd Thesis (I) Development, Implementation and Validation of Operational Concept

1st Concept study „adaptive-augmented outside view“ (2016)

Workplace Analyses & Model Development (2015)

2nd Concept study „integrated information management“ (2017)

Scope of Phd Thesis (II)

3rd Study „low visibility“ and concept transfer on Hololens (2018)

„AR Attention Guidance“ (09/2018)

…work in progress…

Validation of Operational Concept (End 2018)

…work in progress…

Background

Method

Concept

Summary & Outlook

Concept in a nutshell

Adaptive Information Management

presentation of operat. relevant information

reduction of workload and displayclutter

enhanced

situational awareness

Increase capacity and safety under IMC?

Direct interaction

Context

Flight Plan Data

Surveillance Data

Environmental Data

Assistance System

Integrated Information Management

indirect interaction

Ausblick

Background

Method

Concept

Outlook & Summary

Research Aim

Experimental Setup

Results

Conclusion

2nd Concept Study „Flight Data Handling with

Augmented Reality“

LTU7HJ

DLR 360° Apron and Tower Simulator

Research objectives

• Evaluation the current state of the

operational concept

• Identify most promising CWP configuration

• Test the effect of augmented reality based

flight data handling on human performance

Research hypotheses

H1 “An augmented outside view with integrated information management

(H1.1) leads to significantly longer head up times

(H1.2) yields improved situational awareness

(H1.3) yields lower workload

compared to a conventional tower controller working position”

H2 “Direct interaction in the augmented outside view yields

(H2.1) higher situational awareness

(H2.2) lower workload

compared to indirect interaction”

Experimental Setup

[1] TFDPS (tower flight data processing system) [2] Weather Display [3] Air Situation Data Display [4] Ground Radar Display

Configuration C4 „context-adaptive augmented outside view with both interaction possibilities“

Configurations

Configuration Description Technical Setup

“conventional CWP”

experimental tower controller working position (no augmented outside view)

“indirect interaction”

C1 + augmented outside view + coupling with TFDPS

“direct interaction”

C1 + augmented outside view + coupling with 3D “iDrive” Controller. Data entry via context-menu. No TFDPS available!

“both interaction methods”

C3 + augmented outside view + TFDPS: data entry both via TFDPS and 3D controller possible

Configuration C4 „context-adaptive augmented outside view

with both interaction possibilities“

Eyetracking: Areas of Interests

Wearable eye tracking glasses SMI ™ Eye Tracking Analyzer (DLR)

Scenarios and Experimental Procedure

• Hamburg Local Control • IFR Traffic (just few VFR flights) • Arrivals only on RWY 23, Departures from RWY 33 • only VMC • Traffic volume (above medium, rather high) • R/T with 2 Pseudopilots • no telephone coordination with other ATC units

Part I: • 4 simulation runs á 45 minutes (ca. 48 IFR, 2 VFR

flights) • Eyetracking recording during simulation runs • Post-Run Questionnaires & Debriefings

Part II: • Workshop

Test persons (n=4): 3 DFS Aerodrome Controller, 1 Military Aerodrome Controller

Results: Eyetracking

Standard Errors for Head-Up: [C1]: ±1.60 [C2]: ± 4.37 [C3]: ± 3.81 [C4]: ± 5.16 Standard Errors for Head-Down: [C1]: ±2.29 [C2]: ± 2.69 [C3]: ± 1.34 [C4]: ± 2.46

C1 C2 C3 C4

Controller Working Positions

Head-Up

Head-Down

89.7 68.1 43.0 57.7

57.0 42.3

Results: Situation Awareness (3D SART)

C1 C2 C3 C4

Mean SA 51,3 60,3 22,3 72,0

Mean Situational Awareness

Results: NASA-TLX Workload Scores

C1 C2 C3 C4

Overall Workload (all subjects) 8,0 7,6 10,6 6,6

NASA TLX (Weighted)

Background

Method

Concept

Conclusion & Outlook

Conclusion (I)

• configuration C4 showed mean dwell times for head up increased more than 25% compared to a conventional tower CWP (hypothesis H1.1)

• configuration C4 promotes the situation awareness at most (hypothesis H1.2) • configuration C4 NASA TLX workload ratings were the lowest when compared

to the other CWP configurations (hypothesis H1.3) the results support the hypothesis 1 (H1.1, H1.2 and H1.3) almost entirely

In general, configuration C4 was rated as the most beneficial and best-suited CWP for an augmented reality based flight data handling

Conclusion (II)

• Despite the high dwell times for head up configuration C3 (89.3%), direct interaction in the augmented outside view did neither automatically yield higher situational awareness (hypothesis H2.1) than C2 (indirect interaction) (actually the opposite was the case! C3 obtained the lowest S.A. ratings)

• C3 yielded highest workload scores (hypothesis H2.2)

• possibly due to …. absence of TFDPS (important for flight planning) …. fairly high traffic volume in the scenarios, …. insufficient training, unusual working method, …. limitations related to implementation and non- ergonomic handling Thus, hypothesis H2.1 and H2.2 could not be confirmed!

Is “Head Up Only” a desirable approach?

Summary

• Preliminary results although of descriptive nature are promising

• Combined direct/indirect Interaction with AR flight data is the most beneficial and best-suited configuration for an augmented reality based flight data handling

• Substantial increase of head up time when working with direct/indirect Interaction with AR flight data could be a strong indicator for improved situation awareness

But things are not that simple, with a look at configuration C3 There must be some sweet spot for head up

• migration tolerant system development: at early stages of system development,

integration of AR within the operational environment should be done carefully in small steps (thus existing ATC systems such as TFDPS should be incorporated within new introduced assistance systems. At least in the beginning…)

Outlook

• Future work will focus on the enhancement of the interaction concept

• Evaluate the suitability of the operational concept under all-weather operations (3rd study)

• Final Validation Campaign currently under preparation sufficiently large sample size (n=15 ATCOs) will be determined in order to apply inferential statistics and test for significant effects

… and to have fun and to keep me busy with lots of data collecting and analyzing…

Take Home Message

Results encourage a paradigm shift for flight data handling with augmented reality:

away from a display system to an operating system!

www.DLR.de/fl • Folie 32 CONTACT Hejar Gürlük hejar.guerluek@dlr.de +49 531 295 2591 German Aerospace Center (DLR) Institute of Flight Guidance Controller Assistance

Thank you very much for your attention! That‘s all!

Thank you!

flight data handling with augmented reality · flight data handling with augmented reality doctoral...

Documents

mixed reality: virtual reality, augmented reality, augmented...

enable your virtual reality & augmented reality technology...

augmented reality case study augmented reality

augmented experience: augmented reality

the power of augmented reality and virtual reality for...

state of augmented reality, virtual reality and mixed...

treadmill 2 augmented reality (t2 ar) iss flight ... - nasa

ar - tlc augmented reality big2 · augmented reality...

the impact of augmented reality on e-commerce kannaiah...

augmented reality for smartphones - 123seminarsonly.com ·...

augmented reality (ar) - razvoj karijere | center for ... ·...

augmented and virtual reality - massachusetts institute of...

augmented and visual reality. what is augmented and visual...

augmented reality in education at augmented reality planet...

enrich life with augmented reality ui of the future what is...

augmented and mixed reality - oamk.fi · project...

virtual- and augmented reality in education · •augmented...

augmented reality

the emergence of augmented reality in orthopaedic surgery...

augmented reality or augmented space?