cristina conati department of computer science university of british columbia beyond...
TRANSCRIPT
![Page 1: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/1.jpg)
Cristina ConatiDepartment of Computer ScienceUniversity of British Columbia
Beyond Problem-solving: Student-adaptive Interactive Simulations
for Math and Science
![Page 2: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/2.jpg)
Overview
Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges
– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm
Conclusions and Future work
![Page 3: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/3.jpg)
Intelligent Tutoring Systems (ITS)
Create computer-based tools that support individual learners By autonomously and intelligently adapting to their specific needs
StudentModel
Tutor
DomainModel
Adaptive Interventi
ons
![Page 4: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/4.jpg)
ITS Achievements
In the last 20 years, there have been many successful initiatives in devising Intelligent Tutoring Systems
(Woolf 2009, Building Intelligent Interactive Tutors, Morgan Kaufman)
Mainly ITS that provide individualized support to problem solving through tutor-lead interaction (coached problem solving)– Well defined problem solutions => guidance on problem solving
steps– Clear definition of correctness => basis for feedback
![Page 5: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/5.jpg)
Beyond Coached Problem Solving
Coached problem solving is a very important component of learning
Other forms of instruction, however, can help learners acquire the target skills and abilities– At different stages of the learning process– For learners with specific needs and preferences
Our Goal: Extend ITS to other learning activities that support student initiative and engagement: – Interactive Simulations– Educational Games
![Page 6: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/6.jpg)
Overview
Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges
– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm
Conclusions and Future work
![Page 7: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/7.jpg)
Challenges
Activities more open-ended and less well-defined than pure problem solving
– No clear definition of correct/successful behavior
Different user states to be captured (meta-cognitive, affective) in order to provide good tutorial interventions– difficult to assess unobtrusively from interaction events
How to model what the student is doing? How to provide feedback that fosters learning while
maintaining student initiative and engagement?
![Page 8: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/8.jpg)
Our Approach
Student models based on formal methods for probabilistic reasoning and machine learning
Increase information available to student model through innovative input devices:– e.g. eye-tracking and physiological sensors
Iterative model design and evaluation
![Page 9: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/9.jpg)
Overview
Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges
– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm
Conclusions and Future work
![Page 10: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/10.jpg)
ACE: Adaptive Coach for Exploration
Activities organized into units to explore mathematical functions (e.g. input/ouput, equation/plot)
Probabilistic student model that captures student exploratory behavior and other relevant traits
Tutoring agent that generates tailored suggestions to improve student exploration/learning when necessary
(Bunt, Conati, Hugget, Muldner, AIED 2001)
![Page 11: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/11.jpg)
Adaptive Coach for Exploration
EDM 2010
11
![Page 12: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/12.jpg)
12
Adaptive Coach for Exploration
![Page 13: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/13.jpg)
13
Adaptive Coach for Exploration
Before you leave this exercise, why don’t you try scaling the function by a large negative value?Think about how this will affect the plot
![Page 14: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/14.jpg)
ACE Student Model(Bunt and Conati 2002)
Knowledge
Individual Exploration Cases
Exploration of Exercises
Exploration Categories
Exploration of Units
Iterative process of design and evaluation Probabilistic model of how individual exploration actions
influence exploration and understanding of exercises and concepts
e.g. (in Plot unit) • positive/negative slope• positive/negative intercept• large/small, positive/negative exponents…
![Page 15: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/15.jpg)
Modeling Student Exploration
Our first attempt (Bunt and Conati, 2002)
Learning
Student Model
Number and Coverage of Exploratory Actions, e.g.• Positive/negative Y-Intercept• Odd/Even, Positive Negative Exponent....
Interface Actions
![Page 16: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/16.jpg)
Preliminary Evaluation
Quasi-experimental design with 13 participants using ACE (Bunt and Conati 2002)
– The more exercises were effectively explored according to the student model, the more the students improved
– The more hints students followed, the more they learned
Because the model only considers coverage of student actions, it can overestimate student exploration
Need to consider whether the student is reasoning about the effects of his/her actions– Self-explanation meta-cognitive skill:
![Page 17: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/17.jpg)
Revised User Model (Bunt, Muldner and Conati, ITS2004; Merten and Conati, Knowledge Based Systems 2007)
Learning
Student Model
Number and coverage of student actions
Self-explanation of action outcomes
Time between actions Gaze Shifts in Plot Unit
Interface Actions
Input from eye-tracker
![Page 18: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/18.jpg)
Results on Accuracy
We evaluated the complete model against– The original model with no self-explanation– A model that uses only time in between actions as evidence of self-
explanation
Accuracy on SE Accuracy on Learning
50
60
70
80
No SE
SE (Time)
SE (Time + Gaze)
![Page 19: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/19.jpg)
What’s Next (1)
Test adaptive interventions to trigger self-explanation (Conati 2011)
![Page 20: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/20.jpg)
Discussion
ACE work provided evidence that• It is possible to track more “open ended” students’
behaviors than structured problem solving• eye-tracking can support the process
However, hand-coding the relevant behaviors, as we did
for ACE (knowledge-based approach)
• is time consuming
• likely to miss other, less intuitive patterns of interaction
related to learning (or lack thereof)
![Page 21: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/21.jpg)
Alternative Approach (Amershi and Conati 2009, Kardan and Conati 2011)
Behavior Discovery Via Data Mining
Association Rules
MiningClustering
Actions LogsOther Data
Fe
atu
re
Ve
cto
rs
Vector of Interaction Features- Frequency Of
Actions- Latency Between Actions……………
Extract rules describing distinguishing patterns in each cluster
Groups together students that have similar interaction behaviors
Interpret in terms oflearning
• Experts• Performance
Measure(s)
![Page 22: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/22.jpg)
Overview
Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges
– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm
Conclusions and Future work
![Page 23: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/23.jpg)
Tested with AI Space CSP applet
AISpace (Amershi et al., 2007)
– set of applets implementing interactive simulations of common Artificial Intelligence algorithms
– Used regularly in our AI courses– Google “AISpace” if you want to try it out
Applet for Constraint Satisfaction problems (CSP), visualizes the working of the AC3 algorithm
![Page 24: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/24.jpg)
27
AISpace CSP Applet
Direct Arc Clicking
![Page 25: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/25.jpg)
User Study (Kardan and Conati 2011)
65 subjects– Read intro material on the AC-3 algorithm– Pre test– Use CSP applet on two problems– Post test
13,078 actions More than 17 hours of interaction
![Page 26: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/26.jpg)
Dataset
Features:– frequencies of use for each action– pause duration between actions (Mean and SD)– 7 actions 21 features
Performance measure for validation– Learning Gain from pretest to posttest
Feature vectors
Clustering
Behavior Discovery
Rule Mining
![Page 27: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/27.jpg)
Found 2 clusters
Statistically significant difference in Learning Gains (LG)– High Learners (HL) and
Low Learners (LL) clusters
32
Feature vectors
Clustering
Behavior Discovery
Rule MiningClustering
![Page 28: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/28.jpg)
Usefulness:Sample Rules
HL members: Use Direct Arc Click action very frequently (R1).
HL cluster:
R1: Direct Arc Click frequency = Highest (Conf =100%, Class Cov = 100%)
LL cluster:
R2: Direct Arc Click Pause Avg = Lowest (Conf =100%, Class Cov = 100%)
R3: Direct Arc Click frequency = Lowest (Conf = 93%, Class Cov=93.5%)
33
LL members: Use Direct Arc Click sparsely (R3) Leave little time between a Direct Arc Click and the next action
(R2)
Feature vectors
Clustering
Behavior Discovery
Rule Mining
![Page 29: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/29.jpg)
Great, but what do we do with this?
We can use the learned clusters and rules to classify a new student based on her behaviors
Use detected behaviours for adaptive support– Promoting the behaviours conducive of learning– Discouraging/preventing detrimental behaviours
34
![Page 30: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/30.jpg)
The User Modeling Framework
35
Association Rules
MiningClustering
Feature Vector
Calculation
OnlineClassifier
Adaptive Interventions
Behavior Discovery
User Classification
Actions LogsOther Data
F
e
at
u
re
New user’s
Actions
Vector of Interaction Features
If user is a LL and uses Direct Arc Click very infrequently (R3)
Then prompt this action
If user is a LL and pauses very briefly after a Direct Arc Click (R2)
Then take action to slow her down
![Page 31: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/31.jpg)
Classifier Evaluation
Leave-one-out Cross Validation on dataset of 64 users For each user u in dataset
1. Remove user u
2. do Behaviour Discovery on the remaining 63
3. for each of u’s actions:» Calculate the feature vector uv
» Classify uv
» Compare with u’s original label
![Page 32: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/32.jpg)
Accuracy as a function of observed actions
![Page 33: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/33.jpg)
Discussion
User modeling framework for open-ended and unstructured interactions– Relevant behaviours are discovered via data mining
techniques instead being hand-crafted Very encouraging results with CSP applet
– Detected clusters represent groups with different learning gains
– Online classifier: good accuracy soon enough to generate adaptive interventions
– These interventions can be derived from the generated rules
![Page 34: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/34.jpg)
Current Work
Applying the discovered rules to generate the adaptive version of the CSP applet
Adding eye-tracking input to the dataset
![Page 35: Cristina Conati Department of Computer Science University of British Columbia Beyond Problem-solving: Student-adaptive Interactive Simulations for Math](https://reader038.vdocument.in/reader038/viewer/2022110321/56649cfa5503460f949cbf4c/html5/thumbnails/35.jpg)
Conclusions
Research on devising student-adaptive didactic support for exploratory activities beyond problem solvingInteractive simulations
Challenges in modeling interactions with no clear structure or definition of correctness
Student modeling approaches based on probabilistic techniques and unsupervised machine learning very promising results
Shown how eye-tracking can help!We are also exploring it in relation to assessing engagement
and attention in educational games (Muir and Conati 2011)