2015-03-18 research seminar, part 2
Post on 16-Jul-2015
42 Views
Preview:
TRANSCRIPT
IFI Tallinn University | 18.03.2015 1
Studies on Face-to-face CSCL Orchestration Load Using Eye-tracking TechniquesLuis P. PrietoCHILI Lab - EPFL
IFI Tallinn University | 18.03.2015
IFI Tallinn University | 18.03.2015 2
Orchestration??
Background
Sometimes a video is worth a thousand slides...
Orchestration
IFI Tallinn University | 18.03.2015 3
Orchestration & Orchestration Load
Background
What do you mean by orchestration??“process of productively coordinating supportive interventions across multiple learning activities occurring at multiple social levels” [Dillenbourg, Järvelä & Fischer, 2009]
in short: effort to manage a learning situation● for the teacher, in most formal education● we focus on collaborative, technology-
enhanced
Why does it matter?We (researchers) propose new methods, technologies, scripts for more effective learning but… does it work in a classroom of 20? 200? every day?
Orchestration
IFI Tallinn University | 18.03.2015 4
Cognitive Load (CL)
Background Cognitive Load
Cognitive Load: use of the limited cognitive processing capacityUsed a lot in psychology, HCI… and CSCL
Ill-defined and difficult to measure
Things people have tried:● Subjective (e.g., ask in questionnaires)● Dual-task (e.g., simple task during main
task)● Physiological (e.g., heart rate, brain
imaging, eyetracking)
IFI Tallinn University | 18.03.2015 5
The Case of Face-to-face CSCL Orchestration
Background F2F CSCL
How to measure CL in a face-to-face classroom CSCL situation?● Dual-task? Too disruptive● Brain imaging? Not feasible in a classroom (for high quality data)● Subjective? Maybe, but requires interruptions or rely on memory about long (1-
hour) period● ...
Could eye-tracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?● Problem: Eye-tracking studies use very controlled conditions (e.g., lighting)!● Solutions: Mobile eye-trackers, Triangulate among multiple relevant eye-tracking
measures of CL [Buettner, 2013]
IFI Tallinn University | 18.03.2015 6
Research Question and Studies
Research question Study overview
Could eyetracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?
Iterative, incremental exploration
1 2 3
IFI Tallinn University | 18.03.2015 7
Research Question and Studies
Study 1
Could eyetracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?
Iterative, incremental exploration
1 2 3
IFI Tallinn University | 18.03.2015 8
Study 1 (Lab): A Tetris Game → Context and Methods
Study 1 Context & Methods
Task: Tetris game
Subjects: n=16, university students
Data gathering: Eye-tracking + game metrics (time series)
Data analysis: ● 4 eye-tracking measures (Buettner, 2013)
a. Pupil diameter mean (in last 10s)b. Pupil diameter stdev (in last 10s)c. Number of fixations >500ms (in last 10s)d. Average saccade speed (in last 10s)
● … over rolling window of 10s, with 5s of slide● Median cut for the session, add up for a Load Index (0-4, representing confidence
that an episode is high cognitive load)● Evolution of Load Index over each game, Comparison with Tetris game metrics
IFI Tallinn University | 18.03.2015 9
Study 1 (Lab): A Tetris Game → Results
Study 1 Results
One game:
Averaging 128 games
IFI Tallinn University | 18.03.2015 10
Research Question and Studies
Study 2
Could eyetracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?
Iterative, incremental exploration
1 2 3
IFI Tallinn University | 18.03.2015 11
Study 2 (Semi-authentic): A Multi-tabletop Open Day → Context and Methods
Study 2 Context & Methods
Context: Open Day for local schools, 3 sessions with ~20 students each, 5 augmented paper tabletops (fractions activities)Subject: 1 Researcher acting as main facilitator/teacher (+2 researchers as assistants, actual teachers observing)Data gathering: Video recording, Eye-trackingData analysis: Calculation of Load Index (as before), Video coding of extreme load 10s episodes (0 or 4, total 315 episodes)● Three dimensions: Type of facilitation activity (e.g., explanation, monitoring), social
plane (e.g., small group, class-level), main gaze focus (e.g., tabletop, student face)
IFI Tallinn University | 18.03.2015 12
Study 2 (Semi-authentic): A Multi-tabletop Open Day → Results
Study 2 Results
Pearson’s chi-squared test of independenceCognitive load being different along the different video coding dimensions:● Activity: p=0.0016*● Social Plane: p<0.001**● Main gaze focus: p<0.001**
(Videos of high- and low-load episodes)
Which factors contribute more to high- or low-load episodes?● Low load: Focus on tabletop, Small-group plane ● High load: Task transition/distribution, Focus on students’ backs or faces, or
teacher desk (cluttered) with paper manipulatives
IFI Tallinn University | 18.03.2015 13
Research Question and Studies
Study 3
Could eyetracking help us keep track of (teacher’s) cognitive load while orchestrating a CSCL situation?
Iterative, incremental exploration
1 2 3
IFI Tallinn University | 18.03.2015 14
Study 3 (Authentic): University Course With Laptops → Context and Methods
Study 3 Context & Methods
Context: Master-level university course on learning technologies, lecture and group work on visualizations, 3 sessions, ~12 studentsSubject: 1 expert teacher (2 sess), 1 novice teacher/TA (1 sess)Data gathering: Video recording, Eye-trackingData analysis: Calculation of Load Index (as before), Video coding of extreme load 10s episodes (0 or 4, total 242 episodes)● Three dimensions: Type of facilitation activity (e.g., explanation, monitoring), social
plane (e.g., small group, class-level), main gaze focus (e.g., tabletop, student face)
IFI Tallinn University | 18.03.2015 15
Study 3 (Authentic): University Course With Laptops → Results
Study 3 Results
Pearson’s chi-squared test of independenceCognitive load being different along the different video coding dimensions:
Which factors contribute more to high- or low-load episodes?For novice teacher:● Low load: Repairs, Focus on teacher computer● High load: Monitoring, Class-level, Focus on student faces
For expert teacher: Similar trends, but much less significant
Coding dimension Overall Novice teacher Expert teacher
Activity p=0.04* p<0.001** p=0.5
Social plane p<0.001** p<0.001** p=0.24
Main gaze focus p<0.001** p<0.001** p=0.001*
IFI Tallinn University | 18.03.2015 16
Discussion
Discussion
● YES, WE CAN use the eye-tracking methods in authentic classroom conditions● We are able to distinguish clear profiles of (fine-grained) high- and low-load
orchestration episodes, in a more objective manner● This is not THE method, it is A method to add to our existing research toolkit to
study CSCL in authentic settings● Teacher-specific differences, Technology/Task-specific differences, but also
common trends
This is only a first approximation! Limitations:● In each study, n=1 … hardly generalizable (should we start creating and sharing
datasets?)● We capture only variations in load within a session (does not account for
especially easy/difficult sessions● The method still requires a researcher on-site (not for fully in-the-wild studies)
IFI Tallinn University | 18.03.2015 17
Conclusion
Conclusion
● Orchestration of (face-to-face) CSCL is crucial for everyday adoption● Proposed a novel method for (fine-grained) estimation of cognitive load using
eyetracking and post-hoc video coding● Focuses attention of researchers in critical orchestration episodes, and
characterizes them● Main conclusion from 3 exploratory studies: it is feasible, even in authentic
classroom conditions● First insights for CSCL research: class-level orchestration is more challenging,
“reading students” is difficult
IFI Tallinn University | 18.03.2015 18
Next steps
Next steps
● Comparative technology studies (same teacher, similar tasks, different technologies)
● Complement/Compare with subjective methods for measuring CL● “Good” vs. “Bad” CL… can we distinguish?● Understanding orchestration process at different time granularities (ms → hr)
IFI Tallinn University | 18.03.2015 19
… and a shameless plug
A shameless plug
https://sites.google.com/site/occw15/
IFI Tallinn University | 18.03.2015 20
References
References
● Buettner, R. (2013). Cognitive Workload of Humans using Artificial Intelligence systems: Towards Objective Measurement applying Eye-Tracking Technology. In KI 2013: Advances in Artificial Intelligence (pp. 37-48). Springer Berlin Heidelberg.
● Dillenbourg, P., Järvelä, S., and Fischer, F. The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder and S. Barnes, eds., Technology-Enhanced Learning: Principles and Products. Springer, 2009, 3–19.
IFI Tallinn University | 18.03.2015 21
In collaboration with:Kshitij SharmaYun WenPierre Dillenbourg
Special thanks to our participant teachers and students
This research was supported by a Marie Curie Fellowship within the 7th European Community Framework Programme (MIOCTI, FP7-PEOPLE-2012-IEF project no. 327384).
top related