flight data handling with augmented reality · flight data handling with augmented reality doctoral...

32
Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25 th 29th 2018 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29 DLR.de/fl Chart 1 Hejar Gürlük DLR Institute of Flight Guidance Controller Assistance

Upload: others

Post on 14-Jul-2020

5 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Flight Data Handling with Augmented Reality

Doctoral Symposium

ICRAT ´18, Castelldefels, Barcelona (Catalonia)

June 25th – 29th 2018

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29 DLR.de/fl • Chart 1

Hejar Gürlük DLR Institute of Flight Guidance Controller Assistance

Page 2: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 2 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Background

Scope of Thesis

Concept

Summary & Outlook

Study

Page 3: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Safety

Highest incidents and accident rates are related to “Runway” (62%) in contrast to “in flight” (3%) (ICAO Safety Report 2015)

Tower: Errors relate mainly to the field of visual perception major contributing factor for runway incursions (Hilburn, 2004)

0

10

20

30

40

50

Perception Memory DecisionMaking

ResponseExecution

Rule Breaking

Error types related to tower environment

60%

6%

3%

62%

Loss of Control in Flight

Runway

Accidents per flight phase

Accidents Fatalities

www.DLR.de/fl • Chart 3 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 4: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 4

„Head Down vs. Head Up Problem“

Head Down (HD): • Information systems and devices; frequent

usage but scan times not as long as head up (Pinska, 2009)

Head Up (HU): • Outside view

• Crucial for maintaining adequate situational

awareness (Hilburn, 2004)

• Remains most important source of information (Ruffner, 2008)

Tower Controller Working Position

Head Up

Problem: frequent transitions from HU to HD: “…delayed detection and interpretation of visual information…major error source for runway incursions“ (Hilburn, 2004)

Head Down

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 5: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 5

Information Complexity

• Widely distributed head-down displays

• Controllers have to mentally merge the different information sources (HD / HU)

• Increasing information complexity (System Wide Information Management)

This can lead to higher workload in terms of

prolonged times for information acquisition and analysis

Tower Controller Working Position

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 6: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Low Visibility Conditions

decreased capacity (throughput) and increased delay at airports operating under CAT II/IIIA-C

www.DLR.de/fl • Chart 6 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 7: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 7 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Background

Scope of Thesis

Concept

Summary & Outlook

Study

Page 8: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Solution Approach and Aim of the Thesis

• Projection of relevant flight and topological data into the real outside

view by means of augmented reality

• prolongation of head-up times,

• improved situational awareness,

• reduced information search

• maintain capacity (even under low visibility conditions)

safety benefits

• Context-adaptive information presentation: user and task-dependent

presentation of only operationally relevant information

• What to present? When to present? How to present?

• reduction of controller workload (information acquisition and -analysis)

www.DLR.de/fl • Chart 8

Motto and Aim: display the right (amount of) information at the right time!

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 9: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 9

Augmented Reality

Issues: - overclutter - untailored information presentation - missing interaction or feedback - integration of AR into operations unclear - display ergonomics, perceptual issues, registration errors

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Research gap: • operational concept for such an assistance

system (e.g. for different operational conditions) -> HOW TO WORK WITH THIS?

• systematic assessment of operational and human performance benefits

overclutter

Page 10: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Main Research Question of Phd Thesis

What are the human performance and operational

benefits of context-adaptive augmented reality

for air traffic control towers?

www.DLR.de/fl • Chart 10 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 11: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Scope of Phd Thesis (I) Development, Implementation and Validation of Operational Concept

1st Concept study „adaptive-augmented outside view“ (2016)

Workplace Analyses & Model Development (2015)

2nd Concept study „integrated information management“ (2017)

www.DLR.de/fl • Chart 11 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 12: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Scope of Phd Thesis (II)

3rd Study „low visibility“ and concept transfer on Hololens (2018)

„AR Attention Guidance“ (09/2018)

…work in progress…

www.DLR.de/fl • Chart 12 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Validation of Operational Concept (End 2018)

…work in progress…

Page 13: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 13 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Background

Method

Concept

Summary & Outlook

Study

Page 14: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Concept in a nutshell

Adaptive Information Management

www.DLR.de/fl • Chart 14

presentation of operat. relevant information

reduction of workload and displayclutter

enhanced

situational awareness

Increase capacity and safety under IMC?

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Direct interaction

Context

Flight Plan Data

Surveillance Data

Environmental Data

Assistance System

Data

Integrated Information Management

indirect interaction

Page 15: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Ausblick

Background

Method

Concept

Outlook & Summary

Study

www.DLR.de/fl • Chart 15 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Research Aim

Experimental Setup

Results

Conclusion

Page 16: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

2nd Concept Study „Flight Data Handling with

Augmented Reality“

LTU7HJ

www.DLR.de/fl • Chart 16 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

DLR 360° Apron and Tower Simulator

Page 17: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Research objectives

• Evaluation the current state of the

operational concept

• Identify most promising CWP configuration

• Test the effect of augmented reality based

flight data handling on human performance

www.DLR.de/fl • Chart 17 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 18: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Research hypotheses

H1 “An augmented outside view with integrated information management

(H1.1) leads to significantly longer head up times

(H1.2) yields improved situational awareness

(H1.3) yields lower workload

compared to a conventional tower controller working position”

www.DLR.de/fl • Chart 18 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

H2 “Direct interaction in the augmented outside view yields

(H2.1) higher situational awareness

(H2.2) lower workload

compared to indirect interaction”

Page 19: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Experimental Setup

[1] TFDPS (tower flight data processing system) [2] Weather Display [3] Air Situation Data Display [4] Ground Radar Display

www.DLR.de/fl • Chart 19 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 20: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Configuration C4 „context-adaptive augmented outside view with both interaction possibilities“

Configurations

Configuration Description Technical Setup

C1

“conventional CWP”

experimental tower controller working position (no augmented outside view)

C2

“indirect interaction”

C1 + augmented outside view + coupling with TFDPS

C3

“direct interaction”

C1 + augmented outside view + coupling with 3D “iDrive” Controller. Data entry via context-menu. No TFDPS available!

C4

“both interaction methods”

C3 + augmented outside view + TFDPS: data entry both via TFDPS and 3D controller possible

Configuration C4 „context-adaptive augmented outside view

with both interaction possibilities“

www.DLR.de/fl • Chart 20 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 21: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Eyetracking: Areas of Interests

www.DLR.de/fl • Chart 21 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Wearable eye tracking glasses SMI ™ Eye Tracking Analyzer (DLR)

Page 22: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Scenarios and Experimental Procedure

• Hamburg Local Control • IFR Traffic (just few VFR flights) • Arrivals only on RWY 23, Departures from RWY 33 • only VMC • Traffic volume (above medium, rather high) • R/T with 2 Pseudopilots • no telephone coordination with other ATC units

Part I: • 4 simulation runs á 45 minutes (ca. 48 IFR, 2 VFR

flights) • Eyetracking recording during simulation runs • Post-Run Questionnaires & Debriefings

Part II: • Workshop

Test persons (n=4): 3 DFS Aerodrome Controller, 1 Military Aerodrome Controller

www.DLR.de/fl • Chart 22 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 23: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Results: Eyetracking

34,5%

50,8%

58,2%

Standard Errors for Head-Up: [C1]: ±1.60 [C2]: ± 4.37 [C3]: ± 3.81 [C4]: ± 5.16 Standard Errors for Head-Down: [C1]: ±2.29 [C2]: ± 2.69 [C3]: ± 1.34 [C4]: ± 2.46

www.DLR.de/fl • Chart 23 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

C1 C2 C3 C4

Me

an D

we

ll Ti

me

s (i

n %

)

Controller Working Positions

Head-Up

Head-Down

89.7 68.1 43.0 57.7

57.0 42.3

10.3

31.9

Page 24: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Results: Situation Awareness (3D SART)

www.DLR.de/fl • Chart 24 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

34,5%

50,8%

58,2%

C1 C2 C3 C4

Mean SA 51,3 60,3 22,3 72,0

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

3D

SA

RT

Sco

re

Mean Situational Awareness

51.3

60.3

22.3

72.0

Page 25: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Results: NASA-TLX Workload Scores

> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

34,5%

50,8%

58,2%

C1 C2 C3 C4

Overall Workload (all subjects) 8,0 7,6 10,6 6,6

0,0

2,0

4,0

6,0

8,0

10,0

12,0

14,0

16,0

18,0

20,0O

vera

ll W

ork

lad

Sco

re

NASA TLX (Weighted)

8.0

7.6

10.6

www.DLR.de/fl • Chart 25

6.6

Page 26: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Chart 26 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Background

Method

Concept

Conclusion & Outlook

Study

Page 27: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Conclusion (I)

www.DLR.de/fl • Chart 27 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

• configuration C4 showed mean dwell times for head up increased more than 25% compared to a conventional tower CWP (hypothesis H1.1)

• configuration C4 promotes the situation awareness at most (hypothesis H1.2) • configuration C4 NASA TLX workload ratings were the lowest when compared

to the other CWP configurations (hypothesis H1.3) the results support the hypothesis 1 (H1.1, H1.2 and H1.3) almost entirely

In general, configuration C4 was rated as the most beneficial and best-suited CWP for an augmented reality based flight data handling

Page 28: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Conclusion (II)

www.DLR.de/fl • Chart 28 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

• Despite the high dwell times for head up configuration C3 (89.3%), direct interaction in the augmented outside view did neither automatically yield higher situational awareness (hypothesis H2.1) than C2 (indirect interaction) (actually the opposite was the case! C3 obtained the lowest S.A. ratings)

• C3 yielded highest workload scores (hypothesis H2.2)

• possibly due to …. absence of TFDPS (important for flight planning) …. fairly high traffic volume in the scenarios, …. insufficient training, unusual working method, …. limitations related to implementation and non- ergonomic handling Thus, hypothesis H2.1 and H2.2 could not be confirmed!

Is “Head Up Only” a desirable approach?

Page 29: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Summary

• Preliminary results although of descriptive nature are promising

• Combined direct/indirect Interaction with AR flight data is the most beneficial and best-suited configuration for an augmented reality based flight data handling

• Substantial increase of head up time when working with direct/indirect Interaction with AR flight data could be a strong indicator for improved situation awareness

But things are not that simple, with a look at configuration C3 There must be some sweet spot for head up

• migration tolerant system development: at early stages of system development,

integration of AR within the operational environment should be done carefully in small steps (thus existing ATC systems such as TFDPS should be incorporated within new introduced assistance systems. At least in the beginning…)

www.DLR.de/fl • Chart 29 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 30: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Outlook

• Future work will focus on the enhancement of the interaction concept

• Evaluate the suitability of the operational concept under all-weather operations (3rd study)

• Final Validation Campaign currently under preparation sufficiently large sample size (n=15 ATCOs) will be determined in order to apply inferential statistics and test for significant effects

www.DLR.de/fl • Chart 30 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

… and to have fun and to keep me busy with lots of data collecting and analyzing…

Page 31: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

Take Home Message

Results encourage a paradigm shift for flight data handling with augmented reality:

away from a display system to an operating system!

www.DLR.de/fl • Chart 31 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29

Page 32: Flight Data Handling with Augmented Reality · Flight Data Handling with Augmented Reality Doctoral Symposium ICRAT ´18, Castelldefels, Barcelona (Catalonia) June 25th – 29th 2018

www.DLR.de/fl • Folie 32 CONTACT Hejar Gürlük [email protected] +49 531 295 2591 German Aerospace Center (DLR) Institute of Flight Guidance Controller Assistance

Thank you very much for your attention! That‘s all!

Thank you!