2015-03-18 research seminar, part 2

IFI Tallinn University | 18.03.2015 1

Studies on Face-to-face CSCL Orchestration Load Using Eye-tracking TechniquesLuis P. PrietoCHILI Lab - EPFL

IFI Tallinn University | 18.03.2015


Orchestration??

Background

Sometimes a video is worth a thousand slides...

Orchestration


Orchestration & Orchestration Load

Background

What do you mean by orchestration??“process of productively coordinating supportive interventions across multiple learning activities occurring at multiple social levels” [Dillenbourg, Järvelä & Fischer, 2009]

in short: effort to manage a learning situation● for the teacher, in most formal education● we focus on collaborative, technology-

enhanced

Why does it matter?We (researchers) propose new methods, technologies, scripts for more effective learning but… does it work in a classroom of 20? 200? every day?

Orchestration


Cognitive Load (CL)

Background Cognitive Load

Cognitive Load: use of the limited cognitive processing capacityUsed a lot in psychology, HCI… and CSCL

Ill-defined and difficult to measure

Things people have tried:● Subjective (e.g., ask in questionnaires)● Dual-task (e.g., simple task during main

task)● Physiological (e.g., heart rate, brain

imaging, eyetracking)


The Case of Face-to-face CSCL Orchestration

Background F2F CSCL

How to measure CL in a face-to-face classroom CSCL situation?● Dual-task? Too disruptive● Brain imaging? Not feasible in a classroom (for high quality data)● Subjective? Maybe, but requires interruptions or rely on memory about long (1-

hour) period● ...

Could eye-tracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?● Problem: Eye-tracking studies use very controlled conditions (e.g., lighting)!● Solutions: Mobile eye-trackers, Triangulate among multiple relevant eye-tracking

measures of CL [Buettner, 2013]


Research Question and Studies

Research question Study overview

Could eyetracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?

Iterative, incremental exploration

1 2 3



Study 1



1 2 3


Study 1 (Lab): A Tetris Game → Context and Methods

Study 1 Context & Methods

Task: Tetris game

Subjects: n=16, university students

Data gathering: Eye-tracking + game metrics (time series)

Data analysis: ● 4 eye-tracking measures (Buettner, 2013)

a. Pupil diameter mean (in last 10s)b. Pupil diameter stdev (in last 10s)c. Number of fixations >500ms (in last 10s)d. Average saccade speed (in last 10s)

● … over rolling window of 10s, with 5s of slide● Median cut for the session, add up for a Load Index (0-4, representing confidence

that an episode is high cognitive load)● Evolution of Load Index over each game, Comparison with Tetris game metrics


Study 1 (Lab): A Tetris Game → Results

Study 1 Results

One game:

Averaging 128 games



Study 2



1 2 3


Study 2 (Semi-authentic): A Multi-tabletop Open Day → Context and Methods


Context: Open Day for local schools, 3 sessions with ~20 students each, 5 augmented paper tabletops (fractions activities)Subject: 1 Researcher acting as main facilitator/teacher (+2 researchers as assistants, actual teachers observing)Data gathering: Video recording, Eye-trackingData analysis: Calculation of Load Index (as before), Video coding of extreme load 10s episodes (0 or 4, total 315 episodes)● Three dimensions: Type of facilitation activity (e.g., explanation, monitoring), social

plane (e.g., small group, class-level), main gaze focus (e.g., tabletop, student face)


Study 2 (Semi-authentic): A Multi-tabletop Open Day → Results

Study 2 Results

Pearson’s chi-squared test of independenceCognitive load being different along the different video coding dimensions:● Activity: p=0.0016*● Social Plane: p<0.001**● Main gaze focus: p<0.001**

(Videos of high- and low-load episodes)

Which factors contribute more to high- or low-load episodes?● Low load: Focus on tabletop, Small-group plane ● High load: Task transition/distribution, Focus on students’ backs or faces, or

teacher desk (cluttered) with paper manipulatives



Study 3



1 2 3


Study 3 (Authentic): University Course With Laptops → Context and Methods


Context: Master-level university course on learning technologies, lecture and group work on visualizations, 3 sessions, ~12 studentsSubject: 1 expert teacher (2 sess), 1 novice teacher/TA (1 sess)Data gathering: Video recording, Eye-trackingData analysis: Calculation of Load Index (as before), Video coding of extreme load 10s episodes (0 or 4, total 242 episodes)● Three dimensions: Type of facilitation activity (e.g., explanation, monitoring), social

plane (e.g., small group, class-level), main gaze focus (e.g., tabletop, student face)


Study 3 (Authentic): University Course With Laptops → Results

Study 3 Results

Pearson’s chi-squared test of independenceCognitive load being different along the different video coding dimensions:

Which factors contribute more to high- or low-load episodes?For novice teacher:● Low load: Repairs, Focus on teacher computer● High load: Monitoring, Class-level, Focus on student faces

For expert teacher: Similar trends, but much less significant

Coding dimension Overall Novice teacher Expert teacher

Activity p=0.04* p<0.001** p=0.5

Social plane p<0.001** p<0.001** p=0.24

Main gaze focus p<0.001** p<0.001** p=0.001*


Discussion

Discussion

● YES, WE CAN use the eye-tracking methods in authentic classroom conditions● We are able to distinguish clear profiles of (fine-grained) high- and low-load

orchestration episodes, in a more objective manner● This is not THE method, it is A method to add to our existing research toolkit to

study CSCL in authentic settings● Teacher-specific differences, Technology/Task-specific differences, but also

common trends

This is only a first approximation! Limitations:● In each study, n=1 … hardly generalizable (should we start creating and sharing

datasets?)● We capture only variations in load within a session (does not account for

especially easy/difficult sessions● The method still requires a researcher on-site (not for fully in-the-wild studies)


Conclusion

Conclusion

● Orchestration of (face-to-face) CSCL is crucial for everyday adoption● Proposed a novel method for (fine-grained) estimation of cognitive load using

eyetracking and post-hoc video coding● Focuses attention of researchers in critical orchestration episodes, and

characterizes them● Main conclusion from 3 exploratory studies: it is feasible, even in authentic

classroom conditions● First insights for CSCL research: class-level orchestration is more challenging,

“reading students” is difficult


Next steps

Next steps

● Comparative technology studies (same teacher, similar tasks, different technologies)

● Complement/Compare with subjective methods for measuring CL● “Good” vs. “Bad” CL… can we distinguish?● Understanding orchestration process at different time granularities (ms → hr)


… and a shameless plug

A shameless plug

https://sites.google.com/site/occw15/




References

References

● Buettner, R. (2013). Cognitive Workload of Humans using Artificial Intelligence systems: Towards Objective Measurement applying Eye-Tracking Technology. In KI 2013: Advances in Artificial Intelligence (pp. 37-48). Springer Berlin Heidelberg.

● Dillenbourg, P., Järvelä, S., and Fischer, F. The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder and S. Barnes, eds., Technology-Enhanced Learning: Principles and Products. Springer, 2009, 3–19.


In collaboration with:Kshitij SharmaYun WenPierre Dillenbourg

Special thanks to our participant teachers and students

This research was supported by a Marie Curie Fellowship within the 7th European Community Framework Programme (MIOCTI, FP7-PEOPLE-2012-IEF project no. 327384).


Questions? Comments?

2015-03-18 research seminar, part 2

Education

gamesifi tallinn university

university studentsdata

eyetracking studies

eyetracking techniquesluis

eyetracking measures

tetris game context

learning situation

tetris game resultsstudy