affective metacognitive scaffolding and user model augmentation for experiental training simulators:...
DESCRIPTION
The ImREAL project (http://www.imreal-project.eu) is researching how to meaningfully augment and extend existing experiential training simulators. The services developed support self-regulated, goal-, and application-oriented learning in adult training. We present results from a study evaluating a medical interview training simulator that has been augmented by an affective metacognitive scaffolding service and by user modelling exploiting social digital traces. Data from 152 medical students participating in this user trial were compared to the results of a prior trial on an earlier technology version. Findings show that students perceived the learning simulator positively and that the enhanced simulator led to increased feelings of success, less frustration, higher technical flow, and more reflection on learning. Interestingly, this cohort of users proved reluctant to provide their open social IDs to enrich their user models.TRANSCRIPT
Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators:
A Follow-up Study
Immersive Reflective Experience-based Adaptive Learning
8th European Conference on Technology Enhanced Learning (EC-TEL 2013)
Gudrun Wesiak, Adam Moore, Christina M. Steiner, Claudia Hauff, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary
Donohoe, Gordon Power, & Owen Conlan
Overview
- Intro
- ImREAL
- Challenges and Approach
- ETU Simulator
- Simulator Augmentation
- Empirical study
- Methods and results
- Discussion & Conclusions
EC-TEL 2013 2
Intro
- Experiential Training Simulators
- For engaging and motivating learning experiences
- To create situational contexts to practice job-relevant skills
- Adult learning (Andragogy)
- Self-directed, experienced-based, goal-oriented, intrinsically motivating, relevancy-driven (Knowles, 1984)
- Influenced by learner’s physical, digital and social world
- Should adapt to learner’s previous experiences, knowledge, ….
EC-TEL 2013 3
4
ImREAL: Challenge
Simulated environment
Augment existing training simulators by...
Real-world activity modelling
Enriched user modelling
Affective-metacognitive scaffolding service
Develop services for different simulators
Real-world experience
Close the gap
Medical interview training
Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode
where the host has to decide about ordering food for his business guests.
The simulated scenario includes four episodes: Greetings (situations embed arriving
on time, different norms about greetings, first impression, and use of body language),
Dinner (situations embed use of body language and different preferences about food
and drink), Bill (situations embed use of body language and different norms about
payment), Goodbye (situations embed use of body language and different norms
about greetings). Figure 2 illustrates the interface and the interaction features provid-
ed to the leaner to select a response and read/write micro-blogging comments.
The simulator was used by 39 users who attended interactive sessions at learning
technology workshops or responded to invitations sent to learning forums in Europe.
The data was collected during the period 29 Oct 2012 – 15 Jan 2013. Micro-blogging
comments (total of 193) were provided by 25 of the users, and were semantically
augmented with ViewS for IC (see Table 3 for a summary).
Table 3. Summary of the ViewS annotation of the micro-blogging content in the study.
Episode Greetings Dinner Bill Goodbye All
Annotations WNAffect 82 84 18 8 192
Body Language 311 236 100 76 723
Distinct Ontology
Entities
WNAffect 36 36 11 5 57
Body Language 76 63 43 33 106
The output of ViewS was shown to two simulator designers (with background in
Psychology) who were involved in the creation and improvement of the simulated
scenario. The semantic augmentation output was visualised in the form of semantic
maps to enable exploration of both ontologies - WNAffect and Body language-, and
identification of ontology entities that linked to user comments and their location over
Business simulation & Buddy program
Simulated world
Real world
Fore
thought R
efle
ctio
n
Experiential Training Simulator: EmpowerTheUser - ETU
Medical Interview Training:
Two Psychiatric Interview Scenarios Mania & Depression
Basic idea: Watch patient video and select appropriate answer
2Training modes: Assess mode and practice mode
EC-TEL 2013 5
ETU: Assess modePerformance Prediction and Scores
EC-TEL 2013 6
Interview steps (Process)Interview skills
ETU scores
ETU: Practice mode
EC-TEL 2013 7
Visible Decision scores
Coach: feedback on decision path
Thinking prompts and scaffolding service
ETU: Practice mode Meta-cognitive scaffolding (MSS)
EC-TEL 2013 8
ETU: Practice mode Meta-cognitive scaffolding (MSS)
EC-TEL 2013 9
Prompts are based on MAI-ROC (Schraw, 194)Prompts are mapped to different interview situations
ETU: Practice mode:Affective Metacognitive Scaffolding
EC-TEL 2013 10
SAM:Smiley Affect Measurement technology
• Explicit assessment of current emotional state with SAM
• Selection of displayed prompts influenced by users’ emotional state
• Sentiment analysis of digital traces (U-SEM)
• Mappings of prompts to situations as weighted list
Empirical Study: Research Questions
• Simulator Augmentation:
• Is the simulation relevant for users?
• Are the services well integrated?
• Digital Traces:
• Usage, privacy & trust
EC-TEL 2013 12
• AMSS: Are there effect on…
• self-regulated learning?
• users’ learning experience?
• learning performance?
• motivation/affect?
Longterm Methodology:
Berthold et al., EC-TEL 2012
Participants
UT1: 143 TCD medical students, 22.7 years, 54.8% male, 97% real world interview experience (15.4% psychiatric)
UT2: 152 TCD medical students, 22.81 years, 50% male, 99% real-world interview experience (25.6% psychiatric)
• 2011
• Pure Simulator
Baseline
• 2012
• ETU with MSS
User Trial 1• 2013
• AMSS, SAM, U-SEM
User Trial 2
Methods
EC-TEL 2013 13
Methods: Procedure and Measures
EC-TEL 2013 14
Cohort survey (N=95)
Training in the simulator (N=143/152)
Post-sim survey (N=39/ 40)
Real-world interview
Post-int survey (N=7/19)
• Demographics• Social Network use & attitudes
• Log-data (Usage, scores)2 scenarios, 2 modes
• Real-World Relevance• SRL-strategies• Motivation & Affect• Learning Experience• Learning Performance
• Real-world relevance• Motivation• Fb on modes, AMSS, SAM, SN
Results: Log DataSimulator Usage & Training Times
15
Usage (solid lines) and mean training times (dashed) of assess and practice mode
UT1: very few users for Mania
UT2: short times for Depression
Assess vs. Practice:
Assess: more users and longer times
UT1 vs. UT2:
Same no. of users, but increased overall training times (both modes)
ResultsWhich aspects are (not) effected?
No differences in ...
Integration & perception of thinking prompts
Self-regulated learning strategies (QSRL)
Motivation to improve skills, to perform good, to apply what has been learned (post-sim & post-int)
No difference between UT1 & UT2, but change from Baseline
Perceived relevance of simulation
Usability (SUS)
Overall motivation
Overall workload (decreased!)
Deacrease from UT1 to UT2
Motivation to learn more about interviewing (post-sim)
EC-TEL 2013 16
Baseline
UT1/UT2
UT1=UT2
ResultsRelevance for Real-World Interviews
UT2 students feel better prepared for real-world interviews
Perception of relevancy & preparedness unchanged after real-world interview
17
ResultsLearning Experience
Flow (Flow short scale; Rheinberg et al., 2003)
increase in overall flow, technical flow & 7 single items
i.e. stronger involvement into the task, greater feeling of smoothness and fluency, smooth interplay between software components
18
Flow score means on item basis (left fig.) and overall means (and SE) for the FKS, worry, and tech scales (right fig.)
ResultsLearning Experience
Workload: Nasa-TLX (Hart & Staveland,1988)
Lower overall workload in UT2
UT2 users felt less frustrated & had stronger feelings of success
EC-TEL 2013 19
Mean scores (and SE) from the NASA-TX workload scales for the two user trials.
ResultsSelf-Regulated Learning
Reflection notes and AMSS prompts
• 1092 triggers provided (M=15/student) in UT2, 2001 in UT1
• MAI-prompts: 88.4% of 69 answers positive
• Increase in proportion of reflective notes
EC-TEL 2013 20
Table 1. Rounded percentages of content types for entries from the note-taking tools.
Note: Entries can be coded to more than one type, thus percentages may exceed 100.
Users Text entries Position Technical Notes Reflection
UT1 50 107 17 57 16 66
UT2 35 86 1 13 7 93
Results: Log DataETU Scores
ETU: predicted and achieved scores in UT1 and UT2
21
Higher scores in UT2 for introduction, eliciting and planning phase
Mainly underestimation of scores in UT2
ResultsAffect
Affective state reported after simulation: Total affective state scales (TASS, Nicolescu et al., 2006 )
Affect during simulation:
SAM was displayed 352 times (2.23 times / user) – used by 1 person
22Mean TASS scores on 7 subscales plus overall mean score across all scales.
NO
Yes
ResultsSocial Networks
Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn
Nobody (of 152) provided SN-ID in simulator (13 did in cohort survey; 11 fb)
For detailed results: see Moore, 2013 (Umap; LBR)
Reasons:
EC-TEL 2013
23
ResultsSummary
• Changes from pure (BL) to augmented (UT) simulator
↑ Perceived relevance, usability, state motivation
↓ Overall workload
• Changes from MSS (UT1) to AMSS (UT2) simulator
Relevancy: ↑ stronger feeling of preparedness
SRL: ↑ effort (times), feeling of success, reflective notes;
Performance: ↑ ETU-Scores
Learning Experience: ↑ flow↓ decrease in frustration
Motivation: ↓ to learn more about interviewing
EC-TEL 2013 24
Conclusions & Challenges
• Metacognitive scaffolding supports learners on different levels
• Evaluation of user model augmentation
Low user acceptance for exploiting their social digital traces
Need to increase willingness to share SN-IDs
• Study learning activity rather than purely subjective reports
Analysing learners’ real activities with the simulators
• (Non)-invasiveness vs. (un)-visibility
SAM, thinking prompts
EC-TEL 2013 25
Conclusions & Challenges
• (Non)-invasiveness vs. (un)-visibility
SAM now with triggers and presented in main window
EC-TEL 2013 26
NO
YES
Thanks for listening!
EC-TEL 2013 27