Using Pupil Diameter to Measure Cognitive Load

Georgios Minadakis and Katrin LohanHeriot-Watt University

MACS DepartmentEdinburgh

Abstract

In this paper, we will present a method for measuring cogni-tive load and online real time feedback using the Tobii Pro 2eye-tracking glasses. The system is envisaged to be capable ofestimating high cognitive load states and situations, and ad-just human machine interfaces to the user’s needs. The sys-tem is using well-known metrics such as average pupillarysize over time. Our system can provide cognitive load feed-back at 17-18 Hz. We will elaborated on our results of a HRIstudy using this tool to show it’s functionality.

IntroductionIn human-machine interaction information delivery and in-terface design has been researched extensively. User cen-tered design is trying to focus on the users needs and iter-atively develops better or more appropriate interfaces. Thisconcept is challenging and long-term through it’s iterativenature. Interfaces need to be tailored to users as well asadapted to high stakes, hazardous tasks. In this work wepresent a system that will use measurements of cognitiveload through the Tobii Pro 2 eyetracking glasses, which al-lows to measure pupil diameter in real-time.For decades now, it has been established that changes inone’s eyes’ pupils diameter is an indicator of cognitive ac-tivity. In the beginning of the previous century the Germanneurologist Bumke had already recognized that every intel-lectual or physical activity translates into pupil enlargement(Hess 1975).Pupillometry is the measurement of pupil size and reactivity,is a key part of the clinical neurological exam for patientsand evaluates the pupils of patients with the focus on thepupil size. Pupillary responses can reflect activation of thebrain allocated to cognitive tasks. Greater pupil dilation isassociated with increased processing in the brain (Granholmand Steinhauer 2004).Hess and Polt achieved a major milestone for pupillome-try by discovering that showing semi nude photos of adultsto subjects of the opposite sex would cause their pupilsto dilate twenty percent on average (Hess and Polt 1960).This study provided evidence that emotional stimulationcauses enlargement of pupil diameter. This notion was laterexpanded upon to include more cognitive processes, such

Copyright c© 2018, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Figure 1: System setup: The tobii glasses 2 are given to eachof our participants our software records their pupil diameterduring the interaction game with the robot. When the cog-nitive load is perceived as high this can be observed by theexperiment as it is display on a screen during the interaction.as memory and problem solving. Beatty and Kahnemanshowed that storing an increasing number of digits in one’smemory would cause pupillary dilation (Beatty and Kah-neman 1966), while it was also shown by Hess and Poltthat pupil size corresponds with the difficulty of a cogni-tive task (Hess and Polt 1964). More recently, Just and Car-penter (Just and Carpenter 1993) showcased that pupil re-sponses can be an indicator of the effort to comprehend andprocess information. They conducted an experiment whereparticipants were given two sentences of different complex-ities to read while they would measure their pupil diame-ters. Pupillary dilation was larger while readers processedthe sentences that were deemed to be more complicated andmore subtle while they read the simpler one (Just and Car-penter 1993). We believe these findings make the connec-tion between pupillometry and cognitive load theory clear,as they demonstrate that changes in the properties of an el-ement to be processed (e.g. changing the complexity of asentence affects the amount of intrinsic load it will imposeto the reader), cause different pupillary responses. Whilecognitive load can be affected by a large number of factors,we believe that pupillometry offers a responsive signal thatcan potentially provide approximate real-time feedback ofthe users arousal and potentially their cognitive load. Tech-niques in pupillometry have been successfully employed

arX

iv:1

812.

0765

3v1

[cs

.HC

] 2

9 N

ov 2

018

to measure load in past studies like ((Palinko et al. 2010;Klingner 2010),etc.). Pupillometry could be a powerful toolto measure cognitive load but could be affected by other con-founding factors. Online SystemOur system connects the Tobii Pro 2 eye-tracking glassesusing the Tobii Pro Glasses 2 python controller (Tommaso2018) library over wifi with the computer (see figure1). Thenew system provides information on the cognitive load ofthe participant at 17 fps. After a very simple calibrationphase that is taking advantage of the pupillar light reflex(Kun, Palinko, and Razumenic 2012), the system is able totrack the users cognitive load, based on an empirically setthreshold. Our system provided a running average ζ, of pupildiameter:

ζ(d) =

∑Nt=1 dtN

where dt is the current pupil diameter and N the number offrames. It further provides a windowed average:

ζ(d) =

∑15t=1 dt15

where the average is calculated based on the past 15 framesonly. Furthermore, our system provides a running peek esti-mation where we assume that, when the size of the pupil islarger than 70% of the maximum the cognitive load is high.

Human Robot Interactions (HRI) studyWe run a HRI study with our system and two robots the Pep-per robot and the Husky robot. Participants were invited toplay a pairs matching game with robots. The robot would as-sist the participants, giving clues about where the matchingpair would be and comment on the participants performance.The participants received more points when asking the robotfor clues. ParticipantsWe had 31 participants between 25-46 years with a mean ageof 30.9 interacting with either of the robots in either of 2 con-ditions (high or low error rate). We received 8 participants tointeract with the Pepper at a low error rate, 9 participants tointeract with the Pepper at a high error rate, 5 participants tointeract with the Husky at a low error rate and 9 participantsto interact with the Husky at a high error rate.

EvaluationIn our experiment, we have participants discover 6 match-ing pairs from overturned cards and the robot would supportthem with clues, which were either perfectly correct in thelow error rate conditions or not always correct in the higherror rate condition. We run a Pearson correlation betweenamount of questions for support participants asked and thenumber of peaks (maximal cognitive load) and found a neg-ative correlation between the two variables, r = -.328, p =.041. This suggests that if the participants decided to searchwithout help for the matching pairs they were cognitivelymore loaded than if they ask for the matching pair from therobot. Furthermore, we found a negative correlation betweennumber of peaks and session duration , r = -.578, p<= .001.This suggests if participants did not rely on the help from therobot, but therefore needed more attempts to find all 6 pairs,

Figure 2: The pupil diameter of the participant whilst therobot is talking to the participant.

they had a higher cognitive load over the session. This isalso supported by the fact that we found a negative correla-tion between number of tries participants needed to find allpairs and the session duration, r = -.680, p <= .001. Finally,we looked into how participants reacted to the robots greet-ing them. Here we found a negative correlation between thenumber of peaks (maximal cognitive load) and the conditionthey where in, r = -.535, p <= .001. For a deeper investiga-tion of this we looked into one participant interacting withthe Pepper robot in the low error rate condition and foundthat the cognitive load might be preemted by the robot talk-ing to the participant (see Figure 2).

Discussion and ConclusionWe can see that there are correlations suggesting that whenparticipants decided to search without help for the matchingpair they are cognitively more loaded than if they ask for thematching pair. This might be related to the condition theyare in. So if the robot was in a low error rate condition theywere asking more for help than if they were in a high er-ror rate condition. We have not yet investigated this aspectof the data. Furthermore, there might be a correlation withage of the participants and the cognitive load that we havenot investigated yet. Using pupil diameter as a measure forcognitive load might depend on age, as the reaction time ofthe pupil changes during aging. Nevertheless, we found rel-ative stable results in the detection of cognitive load in thepupil diameter and we believe therefore it is a good real-timemeasure for cognitive load. Other measures of cognitive loadare of interest for us as well, like verbal features (e.g pitch,volume or velocity) which have recently investigated here(Lopes, Lohan, and Hastie. 2018). Future work seeks to col-lect more data to establish these results presented are robustwith different participants and that we can generate noveltask. So far, our system shows promising way of detectingcognitive load in HRI but further evaluation and data collec-tion are needed. AcknowledgementsThe authors would like to acknowledge the support ofthe EPSRC IAA 455791 along with ORCA Hub EPSRC(EP/R026173/1, 2017-2021) and consortium partners.

References[Beatty and Kahneman 1966] Beatty, J., and Kahneman, D.1966. Pupillary changes in two memory tasks. PsychonomicScience 5(10):371–372.

[Granholm and Steinhauer 2004] Granholm, E., and Stein-hauer, S. R. 2004. Pupillometric measures of cognitive andemotional processes. International Journal of Psychophysi-ology 52(1):1 – 6. Pupillometric Measures of Cognitive andEmotional Processes.

[Hess and Polt 1960] Hess, E. H., and Polt, J. M. 1960. PupilSize as Related to Interest Value of Visual Stimuli. Science132(3423):349–350.

[Hess and Polt 1964] Hess, E. H., and Polt, J. M. 1964. PupilSize in Relation to Mental Activity during Simple Problem-Solving. Science 143(3611):1190–1192.

[Hess 1975] Hess, E. H. 1975. The tell-tale eye: How youreyes reveal hidden thoughts and emotions. 22–24.

[Just and Carpenter 1993] Just, M. A., and Carpenter, P. a.1993. The intensity dimension of thought: pupillometricindices of sentence processing. Canadian Journal of Ex-perimental Psychology/Revue canadienne de psychologieexperimentale 47(2):310–339.

[Klingner 2010] Klingner, J. M. 2010. Measuring cognitiveload during visual tasks by combining pupillometry and eyetracking. Ph.D. Dissertation, Stanford University.

[Kun, Palinko, and Razumenic 2012] Kun, A. L.; Palinko,O.; and Razumenic, I. 2012. Exploring the effects of sizeand luminance of visual targets on the pupillary light reflex.In Proceedings of the 4th International Conference on Au-tomotive User Interfaces and Interactive Vehicular Applica-tions, 183–186. ACM.

[Lopes, Lohan, and Hastie. 2018] Lopes, J.; Lohan, K.; andHastie., H. 2018. Symptoms of cognitive load in interactionswith a dialogue system. In ICMI Workshop on ModelingCognitive Processes from Multimodal Data.

[Palinko et al. 2010] Palinko, O.; Kun, A. L.; Shyrokov, A.;and Heeman, P. 2010. Estimating cognitive load using re-mote eye tracking in a driving simulator. In Proceedingsof the 2010 symposium on eye-tracking research & applica-tions, 141–144. ACM.

[Tommaso 2018] Tommaso, D. D. 2018. Tobii proglasses 2 python controller, https://github.com/ddetommaso/TobiiProGlasses2_PyCtrl. Web-page.