measuring and modeling shared visual attention jeffrey · pdf filemeasuring and modeling...

Measuring and Modeling Shared Visual Attention

Jeffrey B. Mulligan

NASA Ames Research Center

Workshop on Computational and Mathematical Models in VisionMay 2016

MODVIS 2016 file:///Users/jmulliga/working/public_html/home/presentations/modvis16/modvis16_title.htm

1 of 1 5/20/16, 6:31 AM

https://ntrs.nasa.gov/search.jsp?R=20160008099 2018-05-04T14:47:06+00:00Z

Patrick GontarInstitute of Ergonomics, TUMunchen

 My collaborator     

My collaborator file:///Users/jmulliga/working/public_html/home/presentations/modvis16/patrick.htm

1 of 1 5/20/16, 6:31 AM

Simulated approach toNYC/JFK with multipleproblems

Crews rated by instructor(s)

Objective measures: fuelconsumption, timespent/remaining, etc.

High-level goals: - determine what styles ofcrew coordination lead to successfuloutcomes - use results to inform

 Patrick's experiment     

Patrick's experiment file:///Users/jmulliga/working/public_html/home/presentations/modvis16/pg_exp2.htm

1 of 2 5/20/16, 6:31 AM

training   Haslbeck, A., Gontar, P. & Schubert, E. (2014). How Can Procedures and Checklists Help Pilots in Abnormal Flight Situations?In N. A. Stanton, S. J. Landry, G. Di Bucchianico & A. Vallicelli (Eds.), Advances in Human Aspects of Transportation. Part II(S. 456-461). AHFE International.

Patrick's experiment file:///Users/jmulliga/working/public_html/home/presentations/modvis16/pg_exp2.htm

2 of 2 5/20/16, 6:31 AM

 Eye gaze as a proxy for attention    

Eye gaze as a proxy for attention file:///Users/jmulliga/working/public_html/home/presentations/modvis16/orig_data1.htm

1 of 1 5/20/16, 6:32 AM

Data sorted by "UseCase" (phase of flight) and "Area"

 Format of the "raw" data     

Format of the raw data file:///Users/jmulliga/working/public_html/home/presentations/modvis16/raw_data1.htm

1 of 1 5/20/16, 6:32 AM

 Converting the data back to a time series    

Converting the data back to a time series file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data1.htm

1 of 1 5/20/16, 6:32 AM

82 seconds

 A graphical representation     

A graphical representation file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data3.htm

1 of 1 5/20/16, 6:33 AM

21.8 minutes

 A graphical representation     

A graphical representation file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data7.htm

1 of 1 5/20/16, 6:33 AM

Chi-square approach (thanks to A.Ahumada)

Null hypothesis: observations drawn from same distribution   

Choose time window   

Convert from time series to fixation counts

(using average fixation duration)

Chi-square approach file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq1.htm

1 of 1 5/20/16, 6:33 AM

 Observed fixation durations     

Observed fixation durations file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq2.htm

1 of 1 5/20/16, 6:34 AM

 A 21 second example     

A 21 second example file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq3.htm

1 of 1 5/20/16, 6:34 AM

Probability of next state depends only on current state

 A Markov model of fixation sequences    

A Markov model of fixation sequences file:///Users/jmulliga/working/public_html/home/presentations/modvis16/markov2.htm

1 of 1 5/20/16, 6:34 AM

 Example Markov models     

Example Markov models file:///Users/jmulliga/working/public_html/home/presentations/modvis16/mat_example8.htm

1 of 1 5/20/16, 6:35 AM

Synthetic data experiment details        

10 randomly generated activities   

2 sample records for each activity   

Compare statistics of matched/unmatched pairs

Synthetic data experiment details file:///Users/jmulliga/working/public_html/home/presentations/modvis16/synth_exp1b.htm

1 of 1 5/20/16, 6:35 AM

1.4sec

25sec

426sec

 A scale-space image of Chi-squared p    

A scale-space image of Chi-squared p file:///Users/jmulliga/working/public_html/home/presentations/modvis16/scale_space_si...

1 of 2 5/20/16, 6:35 AM

1.4sec

25sec

426sec

 A scale-space image of Chi-squared p    

A scale-space image of Chi-squared p file:///Users/jmulliga/working/public_html/home/presentations/modvis16/scale_space_si...

1 of 2 5/20/16, 6:35 AM

 Histograms of p-values     

Histograms of p-values file:///Users/jmulliga/working/public_html/home/presentations/modvis16/hist_example1.htm

1 of 1 5/20/16, 6:36 AM

 Cumulative distributions of p-values    

Cumulative distributions of p-values file:///Users/jmulliga/working/public_html/home/presentations/modvis16/cum_example1.htm

1 of 1 5/20/16, 6:36 AM

  An ROC curve     

An ROC curve for discriminating matched from mismatched activities file:///Users/jmulliga/working/public_html/home/presentations/modvis16/roc_example1.htm

1 of 1 5/20/16, 6:36 AM

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/single_auc_plot.htm

1 of 1 5/20/16, 6:36 AM

 Between-activity correlations     

Between-activity correlations file:///Users/jmulliga/working/public_html/home/presentations/modvis16/asymp_dist.htm

1 of 1 5/20/16, 6:37 AM

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:37 AM

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:37 AM

  Human data     

Human data file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_auc_plo...

1 of 1 5/20/16, 6:37 AM

 Average of all crews     

Average of all crews file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_auc_plo...

1 of 1 5/20/16, 6:38 AM

What to make of the human data?        

Very low p values at long time windows   

Suggests pilot and first officer average behaviors differ   

Suggests simple Markov model is a poor description

What to make of the human data? file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_puzzle1.htm

1 of 1 5/20/16, 6:38 AM

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:38 AM

Chi-square statistics can be used to compare scan paths 

Results suggest that shared attention in the cockpit

is limited to intervals < 1 minute 

Human activities in the cockpit are poorly described

by random Markov processes 

Real activities are likely to involve small numbers of AOIs 

THANK YOU!

 Conclusions     

Conclusions file:///Users/jmulliga/working/public_html/home/presentations/modvis16/modvis16_conc...

1 of 1 5/20/16, 6:39 AM