mw2012 eyetracking

Evaluating the Practical Applications

of Eye Tracking in Museums

Evaluating the Practical Applications

Ed Bachta & Silvia Filippini-Fantoni

Exploring Visitors’ Engagement

With Artworks

Research results

Research so far has indicated that visitors spend little timelooking at artworks:

• Hein, 1998

• Smith, 2001

• Worths, 2003 (‘grazing’)

• Viewing Project, IMA 2012

• Use of observational rubric is labor-intensive and not always precise

• Eye tracking technology has the potential of being more precise and less time-consuming

• More recent development of less intrusive devices (non head mounted)

Observation vs. eye tracking

Photo from Milekic MW 2010

• Gauge the practicality of using such devices in a museum gallery setting.

• Assess the ability of current eye tracking technology to reveal what visitors are looking at and for how long.

• Explore the potential use of this equipment in a practical setting (e.g. VTS discussion)

Sparks! grant objectives

Infrared Emitters

Camera

EyeTech VT2

• Eye tracker range limitations.

• Too much variation in height when the person is standing.

• For the experiments the viewer has to be seated, with the tracker placed in a fixed position between the tracker and the painting.

Device initial testing

Experiment 1© Edward Hopper.

• Distinguish when the participant looks inside/outside of the painting.

• Measure time spent looking inside/outside of the painting.

• Track where looking inside the field of the artwork.

Calibration performed once

Experiment 1: objectives

The device was installed on a cart between the work of art and the seated participant and calibrated to the first participant.

Participants’ standing and seated height were measured and distance from mid-eye to floor.

• 22 participants were asked to look in and outside the painting for 1 minute.

• First 10 participants could not adjust their chair position to optimize eye tracking, while the next 12 were asked to do so.

• Participants’ gazes (inside the field of the painting) were tracked by 2 research assistants with stopwatches.

• The times were averaged and compared to the time tracked by the device

Experiment 1: part 1

• A subset of participants (8 of the 22) were asked to look (over a period of 60 seconds) at 6 different areas of the work for 10 seconds each in sequence prompted by a research assistant.

• Tracker data was logged in the same manner as the previous experiment.

Experiment 1: part 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22-80

-70

-60

-50

-40

-30

-20

-10

0

10

20

diffe

renc

e (%

of s

essi

on ti

me)

Relative quantity of valid gaze dataMissing data for >25% of session time for 6 participants

Fixed seat position Adjusted seat position

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22-80

-70

-60

-50

-40

-30

-20

-10

0

10

20

diffe

renc

e (%

of s

essi

on ti

me)

Relative quantity of valid gaze dataPoor performance for 4 of 6 participants at low eye level (<50”)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22-80

-70

-60

-50

-40

-30

-20

-10

0

10

20

diffe

renc

e (%

of s

essi

on ti

me)

Relative quantity of valid gaze dataMissing > 10% for 7 of 10 glasses wearers

Comparison against manual measurement

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 220

102030405060708090

100

Participant

erro

r (%

of s

essi

on ti

me)

Comparison againstmanual measurement

Fixed seat position Adjusted seat position

Within 5% 0 5

Within 10% 1 7

• Allowing the participant to adjust the seat position produces better results

• We were still hoping for better accuracy

Gap Handling

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 220

102030405060708090

100

Raw100ms500ms1s

Participant

erro

r (%

of s

essi

on ti

me)

Gap Handling

• Applying a gap algorithm appears to improve results with a 500ms threshold

• Accuracy was within 10% for two thirds of participants when allowing the seat position to be adjusted and using the algorithm

Raw 100ms 500ms 1s

Within 5% (fixed) 0 1 1 1

Within 10% (fixed) 1 3 3 4

Within 5% (adjusted) 5 5 8 7

Within 10% (adjusted) 7 7 8 8

Gaze locations (calibrated)

Gaze locations (calibrated)

Gaze locations (typical uncalibrated)

• Device was not able to continuously track the gaze of a seated viewer.

• An attempt to improve results by handling gaps in the data were successful, but only to a degree.

• Vertical gaze location was not accurate for uncalibrated viewers.

Experiment 1: Summary

Experiment 2

• Measure whether the device could be more precise when calibrated for each participant

• We repeated experiment 1 (part 1 and 2) with 12 participants but calibrated the devices individually

• This second experiment was set in a lab, where the exact size of the painting was reproduced on a board

Experiment 2: objectives& methodology

Relative quantity of valid gaze data

1 2 3 4 5 6 7 8 9 10 11 12

-20

0

20

40

60

80

100

diffe

renc

e (%

of s

essio

n tim

e)

Gaze DurationDelta

(% of session time)Max. calibration

“score”Glasses Seated eye

elevation

7.719 6.70 did without glasses 4915.159 2.76 yes 51.56.306 13.90 no 546.728 6.13 yes 525.164 3.30 no 51.25

10.071 6.78 no 480.007 5.13 no 53.756.082 4.07 did without glasses 47.754.572 4.36 yes 50.52.732 10.10 no 52

12.240 12.15 yes 49.53.790 3.77 no 51

Gaze DurationAccuracy Raw data 100ms threshold 500ms threshold

Within 2% 1 4 6

Within 5% 4 7 8

Within 10% 9 12 12

• An improvement over the first experiment

• One third of participants were in the 5-10% range

Gaze LocationBest session

Gaze LocationWorst session

ComparisonsAverage error

(degrees of FOV)Max. calibration “score” Glasses

0.96 6.70 no1.60 2.76 yes1.72 13.90 no2.47 6.13 yes0.88 3.30 no1.61 6.78 no0.90 5.13 no1.24 4.07 no1.18 4.36 yes1.76 10.10 no3.00 12.15 yes2.22 3.77 no

• Applying the gap handling algorithm brought all sessions within 10% of the manual measurement

• Gaze duration results were better than in the uncalibrated study, but still not what we hoped for

• Gaze location results were also better than in the uncalibrated study, but not as accurate as expected

Experiment 2: Summary

• Experiment 3 will make use of the tracker during a VTS session

• We will evaluate whether the data recorded assists in understanding what VTS participants look at during a session

Future Work

Photo from PAAM.org

Thank You

ebachta@imamuseum.orgsfantoni@imamuseum.org