Revolving Brain

PPIG 2015Evaluation of Familiarity and Mental Workload in Human Computer Interaction With Integrated Development Environments using Single-Channel EEGOluwatoyin Fakoredeofakorede@bournemouth.ac.ukShahin Rostamisrostami@bournemouth.ac.ukAlex Shenfielda.shenfield@shu.ac.ukStephen Sigurnjakssigurnjak@uclan.ac.ukPresentation OverviewThe ChallengeWhy EEGHow does EEG relates to HCIThe ExperimentResultsConclusionQuestions?

2The Challenge

Advances and extensive adoption of technology in the workplace have resulted in more information being presented to people.This has led to a higher cognitive demand for processing of the information relevant to the task being performed.This increased cognitive demand and large amount of information can cause confusion, impede comprehension, lead to mistakes or evenCause mental stress.3The Challenge

Perceptual Processor

Cognitive Processor

Motor ProcessorLONG TERM MEMORYWORKING MEMORYAUDITORY IMAGE STOREVISUAL IMAGE STOREThe human mind is an information-processing system. Stuart, Thomas and Allen in their book The Psychology of Human-Computer Interaction Created a model called the Model Human Processor which is divided into 3 interacting subsystems.1) The Perceptual System, 2) Cognitive system and 3) Motor System

1)The Perceptual system receives information from the visual and auditory sensors and stores them in the visual or auditory image store in the short term memory2) Cognitive system which receives information from the sensory image store in its working (short term) memory and uses informationPreviously stored in Long term memory to make decisions on how to respond. The decision is stored in the working memory.3) The Motor system takes the decision from the working memory and channels it to the required part of the body to execute the instruction. The motor system is responsible for carrying out responses.

4The ChallengeEvaluation Methods to Monitor HCI and UI QualityBehavioral studiesCompletion timesError rates

InquiriesQualitative feedback from questionnairesInterviews

Physiological measuresMouse movementEye motions and gazeHeart rates

NeuroimagingSensors which records brain activity

Only Neuroimaging measures cognitive processing effort Methods have been developed to monitor the coordination of these 3 processors/subsystems with respect to HCI and UI qualityThese methods monitor performance such as Completion times, error rates and qualitative feedback from questionnaires, all of these are observable measures. Another method is Physiological measures which are mouse movement, eye motions and heart rates.The first three are observable methods which measures perceptual and cognitive system but the cannot measure the cognitive processing effort.Only Neuroimaging measures the cognitive processing effort. 5Why EEG?Neuroscience researchExpensive to conduct

Electroencephalography (Single-channel dry EEG)Less expensiveLess restrictiveWidely used in research and clinical studiesSafe for extended use

Most neuroscience research on interaction and learning has focussed on imaging the brain using MRI (Magnetic Resonance Imaging) scans. This works by providing the exact positioning of the areas of activation in the brain and their respective durations .Our approach (Single-channel dry EEG) is cheaper and more flexible (100)6Why EEG?Research HCI issuesMedical applicationsAdaptive user interfacesMeasure of mental effort and familiarity of Participants when tracing shapes using their Non-dominant hand.EEG Applications

Most neuroscience research on interaction and learning has focussed on imaging the brain using MRI scans. This works by providing the exact positioning of the areas of activation in the brain and their respective durations.The downside of this is that its very expensive to conduct, specialist equipment is required and its restrictive in thatThe person is not free to perform the experiment as they would in the workplace, nor everyday activities. 7Why EEG?Neurosky Mindwave headsetEEG Device

Nuerosky Mindwave headset is ergonomic, minimally intrusive, lightweight, and single-channel dry sensor EEG 8EEG and HCIMental Workload AttentionVigilanceFatigueFamiliarityError RecognitionUsers State Patterns

Users State Patterns are used to characterize Human Computer Interactions. In other words, the different cognitive states that can be assessed using neuroimaging techniques. Mental workload: During the processing of information, there is a relationship between performance and processing power from HCI perspective.Workload is the ratio between processing power and performance (or data coming from the environment). If the mental workload is too high the Subjects performance decreases. Workload is directly proportional to processing power and inversely proportional to performance.

Attention refers to the ability to focus cognitive resources on a particular stimulus.Insufficient attention result in difficulty or inability to perform the task

Vigilance refers to a state of sustained attention. One needs to maintain a high degree of vigilance overtimes in order to focus his attention on something.Alertness would be a synonym of vigilanceFatigue is a state in which cognitive resources are exhausted. If the required level of vigilance or attention causes a strain, then fatigue arises and performances Decrease. Then the task cannot be performed correctly or errors appear.Familiarity is a measure of how well a subject is learning a specific taskError recognition is a situation when the users detect by themselves an outcome different from what is expected.

EEG can be used to measure these states.

912 volunteersAll reported to have normal or corrected-to-normal visionNo known history of neurological or physiological disorder Participants were reported to have not used the Integrated Development Environment (IDEs) previously

The ExperimentEach participant performed eight trials involving HCIEither of two software IDEs was consideredThe participants were required to follow clearly defined instructions to complete tasksThe IDEs considered were Visual Studio by Microsoft and Eclipse by the Eclipse FoundationTwo shell scripts were developed and executedTwo computers were used during the experimentTwo facilitators were involved.

The ExperimentData AcquisitionSystem Architecture Diagram here

The ExperimentData AcquisitionTask Instructions

A simple data logging application was developed on top of the API provided by NeuroSky. This API provides a simple interface for connecting to the EEG headset and acquiring data to implement mental effort and task familiarity algorithms 12The ExperimentData AcquisitionUML activity diagram

A UML activity diagram for the data logging application used in this research is providedin here showing the work-flow through the application. One of the key activities withinthis work-flow is the initial calibration of the mental effort and task familiarity metrics. Thisinvolves the subject relaxing with eyes open for 60 seconds before starting the task to allowfor the calculation of initial baseline values for both mental effort and task familiarity. Thesebaseline values can then be used as a reference for comparing later values. New values for mentaleffort and task familiarity are calculated continuously every 10 seconds. It should also be notedthat the application also checks the signal quality every second to ensure that the headset isconnected and positioned properly - if a poor signal quality indicator is detected then no datais collected and the trial facilitator is notified so that they can help readjust the headset.13The ExperimentData AcquisitionGraphical User Interface for the data acquisition application

Observations made by a facilitator were recorded in conjunction with the collection of EEGdata during each task for every participant. These observations included noting:1. When a participant had made an error.2. When a participant had stopped referring to the exercise sheet.3. When a facilitator was required to intervene in the task.4. Comments made by the participant throughout the experiment.Also in conjunction with the collection of EEG data, the participant was asked to completea questionnaire after completing each of the eight tasks. This questionnaire would simply ask:1. Was this task difficult? Answers: Not difficult; Indifferent; Difficult.2. Did you make any mistakes? Answers: Yes; No.3. Did the facilitator intervene? Answers: Yes, No.

The monitor here captures two different scales one for Mental Effort and the second for familiarity14Result 1Plots of the post-processed Mental Effort data over time obtained from the EEG head-set for participants using the Visual Studio IDE

Plots showing Mental Effort data over time obtained from the EEG head-set for participants using Visual Studio ID.There are 8 sets of tasksRed lines indicate an error observed by the facilitator.Dashed lines indicate a perceived error by the participantShaded backgrounds indicate the task from which a participant stopped referring to the exercise sheetFinal row indicates mean average of task data, linear best fit and gradient of slope.The gradient of slope indicate the common trend. Task 1, mental effort increased because lots of processing power is being used, performance is low and lots of errors.Task 4, mental effort decreased, performance has improved and no errors encountered

15Result 2Plots of the post-processed Familiarity data over time obtained from the EEG head-set for participants using the Visual Studio IDE

Plots showing Familiarity data over time obtained from the EEG head-set for participants using Visual Studio ID.There are 8 sets of tasksRed lines indicate an error observed by the facilitator.Dashed lines indicate a perceived error by the participantShaded backgrounds indicate the task from which a participant stopped referring to the exercise sheetFinal row indicates mean average of task data, linear best fit and gradient of slope.The gradient of slope indicate the common trend. Task 1, familiarity dropped and that is indicated by the errors made by the participantsTask 3, familiarity begins to increase

16Result 3Plots of the post-processed Mental Effort data over time obtained from the EEG head-set for participants using the Eclipse IDE

Plots showing Familiarity data over time obtained from the EEG head-set for participants using Visual Studio ID.There are 8 sets of tasksRed lines indicate an error observed by the facilitator.Dashed lines indicate a perceived error by the participantShaded backgrounds indicate the task from which a participant stopped referring to the exercise sheetFinal row indicates mean average of task data, linear best fit and gradient of slope.The gradient of slope indicate the common trend. Task 1, mental effort increased because lots of processing power is being used, performance is low and lots of errors.Task 4, mental effort decreased, performance has improved and no errors encountered

17Result 4Plots of the post-processed Familiarity data over time obtained from the EEG head-set for participants using the Eclipse IDE

Plots showing Familiarity data over time obtained from the EEG head-set for participants using Visual Studio ID.There are 8 sets of tasksRed lines indicate an error observed by the facilitator.Dashed lines indicate a perceived error by the participantShaded backgrounds indicate the task from which a participant stopped referring to the exercise sheetFinal row indicates mean average of task data, linear best fit and gradient of slope.The gradient of slope indicate the common trend. Task 1, familiarity dropped and that is indicated by the errors made by the participantsTask 3, familiarity begins to increase18Conclusion

Eclipse users finished quicker than Visual Studio usersEclipse users made fewer errors than Visual Studio usersVisual Studio IDE participants took longer to become Familiar with the interfaceCorrelation between mental effort and familiarity data withObservation made by both the facilitator and participantLow-cost EEG for evaluating HCI in terms of familiarity andMental effort is feasibleDrawback includes unpredictable behavior of participants, For example, stop referring to the exercise sheetsReferences/ Further reading Antonenko, P., Paas, F., Grabner, R., & van Gog, T. (2010). Using electroencephalography to measure cognitive load. Educational Psychology Review, 22 (4), 425 - 438. Chu, K., & Wong, C. Y. (2014). Player's attention and meditation level of input devices on mobile gaming. In User science and engineering (i-user), 2014 3rd international conference on (pp. 13{17). Cutrell, E., & Tan, D. (2008). BCI for passive input in HCI. In Proceedings of chi (Vol. 8, pp.1-3). Imamizu, H., Miyauchi, S., Tamada, T., Sasaki, Y., Takino, R., PuEtz, B., . . . Kawato, M.(2000). Human cerebella activity reflecting an acquired internal model of a new tool. Nature, 403 (6766), 192-195. Kitamura, Y., Yamaguchi, Y., Hiroshi, I., Kishino, F., & Kawato, M. (2003). Things happening in the brain while humans learn to use new tools. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 417 - 424).References/ Further reading Lee, J. C., & Tan, D. S. (2006). Using a low-cost electroencephalograph for task classifation in HCI research. In Proceedings of the 19th annual ACM symposium on user interface softwareand technology (pp. 81{90). Mak, J. N., Chan, R. H., &Wong, S. W. (2013a). Evaluation of mental workload in visual-motor task: Spectral analysis of single-channel frontal EEG. In Industrial electronics society, IECON 2013-39th annual conference of the IEEE (pp. 8426 - 8430). Mak, J. N., Chan, R. H., & Wong, S. W. (2013b). Spectral modulation of frontal EEG activities during motor skill acquisition: Task familiarity monitoring using single-channel EEG. In Engineering in medicine and biology society (EMBC), 2013 35th annual international conference of the IEEE (pp. 5638 - 5641).Oluwatoyin Fakoredeofakorede@bournemouth.ac.ukFaculty of Science and Technology,Bournemouth University

Shahin Rostamisrostami@bournemouth.ac.ukFaculty of Science and Technology,Bournemouth UniversityAlex Shenfielda.shenfield@shu.ac.ukFaculty of Arts, Computing , Engineering & Sciences, Sheffield Hallam University

Stephen Sigurnjakssigurnjak@uclan.ac.ukSchool of Computing, Engineering & Physical Sciences, University of Central LancashireContactsMore Information?

Thank you, any questions?