jose carlo a. soriano. break down effective help-seeking behavior among students using an...
TRANSCRIPT
Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System
for Math: a Cross-Cultural ComparisonJose Carlo A. Soriano
Break down
Effective Help-seeking Behavior Among Students
Using an Intelligent Tutoring System for Math:
A Cross-Cultural Comparison
Intelligent Tutoring SystemsSeeks to simulate the effectiveness of a good
personal human tutorIndividual tutoring is more effective than
classroom instruction by 2 standard deviations (Bloom, 1984) Knows what specific skills the student is having trouble
with Able to offer appropriate help at an appropriate time
Objective: help the student learnITS are better by 1 standard deviation (Koedinger
et al, 1998; Corbett et al., 2001)
Intelligent Tutoring Systems
Help-Seeking in ITSThe kind of help, and how the help is offered,
affects learning (Aleven et al, 2003)Students who use High-level help most
frequently have the least learning(Matthews and Mitrovic, 2008)
Help-seeking is a Meta-cognitive skillMeta-cognition is “cognition about cognition”
One’s knowledge of the processes in play One’s active control of it during learning
Help-seeking Behavior1. Both the EU and UNESCO declared:
developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007)
2. Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991)
3. Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)
Effective Help-SeekingStudent is more likely to learn when:
Student seeks for help when encountering personal difficulties Student knows what kind of help is needed such
that student can work effectively on his/her own Student knows how to ask for help
Student is not dependent on helpStudent spends time understanding help given
Help-seeking in ITSHowever, students generally do not know
when they need help (Aleven and Koedinger, 2000)
Students “game the system” (Baker et al)
Meta-cognitive tutors have been developed by Aleven et al
“Scooter the tutor” developed by Baker et alTo teach students “how to learn”
The Problem
Is “effective help-seeking” the same across cultures?Very few cross-cultural comparisons
Comparing ITS use between USA and Latin American students(Ogan et al, in press)
Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010)
Implications on Meta-cognitive tutors
Method
Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly differentMight encourage more cross-cultural
comparisonsImplications on future efforts on meta-
cognitive tutoring
Scatterplot tutor
Data
Costa Rica
Mexico
USA
Philippines
Feature Engineering1. Helpavoidance2. Nothelpavoidance3. Helpnonuse4. Unneededhelp5. BugmsgLongpause6. BugmsgShortpause7. HintmsgLongpause8. HintmsgShortpause
Feature Engineering9. HintmsgLongpauseCorrect10. HintmsgShortpauseCorrect11. NothelpavoidanceShortpause12. NothelpavoidanceLongpause13. UnneededhelpShortpause14. UnneededhelpLongpause15. ShortpauseHintmsg16. LongpauseHintmsg17. FirstattemptHintmsg
Feature OptimizationMost features require a threshold, either p-
know or a time threshold
Brute-force grid search:For p-know thresholds, grid-size is 0.05For pause thresholds, grid-size is 0.5 secondsSingle-parameter linear regression for each
threshold for each feature in grid
Feature SelectionCross-validated r was used as the goodness
criterionCorrelation between the predicted learning
values and the actual learning value
The threshold with the best cross-validated r becomes the threshold for each feature
As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation
Model Creation and Evaluation
Models were created using Forward Selection
Models were evaluated by applying each country’s model to each country’s data set
A model were created after combining the four data sets
Brute-Force Grid SearchFeature cut-off r Feature cut-off RCR PH
Helpavoidance 0.15 0.081 Nothelpavoidance 0.4 0.087Helpnonuse 0.15 0.012 Helpnonuse 0.95 0.043
Unneededhelp 1 0.006 NothelpavoidanceShortpause 1 0.108BugmsgLongpause 25.5 0.06 NothelpavoidanceLongpause 0 0.075HintmsgLongpause 47.5 0.054 UnneededhelpShortpause 0.5 0.061HintmsgShortpause 0.5 0.017 US
HintmsgLongpauseCorrect 41.5 0.294 Helpavoidance 0.25 0.122NothelpavoidanceLongpause 45.5 0.284 Helpnonuse 1 0.014
UnneededhelpShortpause 13 0.019 Unneededhelp 1 0.117UnneededhelpLongpause 0 0.008 BugmsgLongpause 57 0.131
MX BugmsgShortpause 0.5 0.026Helpnonuse 1 0.201 HintmsgLongpause 0.5 0.003
BugmsgShortpause 2.5 0.044 HintmsgShortpause 6.5 0.02HintmsgShortpauseCorrect 0.5 0.025 HintmsgLongpauseCorrect 1 0.039
NothelpavoidanceLongpause 58.5 0.018 HintmsgShortpauseCorrect 4 0.073ALL NothelpavoidanceShortpause 0.5 0.265
Helpavoidance 0.05 0.023 NothelpavoidanceLongpause 0 0.096Nothelpavoidance 0.4 0.024 UnneededhelpShortpause 12 0.203
Helpnonuse 0 0.094 UnneededhelpLongpause 0 0.123BugmsgShortpause 2.5 0.071 ShortpauseHintmsg 32.5 0.026
HintmsgLongpauseCorrect 1 0.022 LongpauseHintmsg 37.5 0.04HintmsgShortpauseCorrect 3 0.062
NothelpavoidanceShortpause 20 0.027NothelpavoidanceLongpause 1 0.052
UnneededhelpShortpause 35 0.04UnneededhelpLongpause 1 0.062
AnalysisFirstattemptHintmsg – only feature that was not
able to pass in any countryDifferent nature of skills
Some can be hard at the very start, some can be easily understood from the start
Most skills may require a large number of actions
NothelpavoidanceLongpause – had positive cross-validated correlation in all five data setsIn contrast to theory, this feature has negative
directionality for most countries This might be because students who exhibit the behavior
simply do not understand the skill
Forward SelectionCountry Learning = Cross-validated r
CR
0.132 * Helpavoidance(0.15)+ 7.385 * HintmsgLongpause(47.5)
- 9.096 * HintmsgLongpauseCorrect(41.5)- 21.847 * NothelpavoidanceLongpause(0.25, 45.5)
+ 53.010
0.462
MX
- 0.147 * Helpnonuse(1) - 0.754 * BugmsgShortpause(2.5)
+ 1.187 * HintmsgShortpauseCorrect(0.5)+ 40.652
0.229
PH 0.021 * Helpnonuse (0.95)
- 0.763 * NothelpavoidanceShortpause(0.4, 1)+ 32.423
0.126
US
- 1.021 * Nothelpavoidance(0.25) - 2.870 * BugmsgLongpause(57)
- 6.680 * NothelpavoidanceShortpause(0.25, 19.5) + 5.605 *LongpauseHintmsg(37.5)
+ 12.086
0.350
ALL
0.036 * Helpavoidance(0.05) + 0.082 * Helpnonuse(0)
- 0.491 * BugmsgShortpause(2.5) - 46.861
0. 153
AnalysisHelpavoidance – negative directionality for
CR and PHreinforcces Aleven et al’s findings that avoiding
help is negatively correlated to learning (Aleven et al., 2006)
ButmsgShortpause and NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US.Possibly effect of not spending enough time to
understand bug or hint message provided
Cross-Cultural Evaluation
Country CR MX PH US ALL
CR 0.534 -0.285 0.051 0.151 0.08
MX 0.04 0.392 -0.086 -0.009 0.088
PH 0.004 -0.174 0.203 0.146 0.038
US -0.085 -0.164 0.228 0.476 0.057
All -0.032 0.165 0.047 0.142 0.215
Rows are models, columns are data sets applied to
AnalysisEighteen out of 25 model applications
produced positive correlation between model’s predicted learning and the actual learningNegative correlation means model did worse
than chance at predicting learningLOOCV r values are very high compared to r
when applied to other countriesReinforces hypothesis that help-seeking might
not transfer across countries
AnalysisMX and US – performance on each other’s
data sets are low (-0.009 and -0.164)This means that our model of effective help-
seeking is not effective when applied to the other country
Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students Collaborative nature of students from Mexico may
be the reason why help-seeking is different The help they ask from the tutor will only be help
that they do not get from other students
AnalysisCR and MX – did not perform well on each other’s data
sets (-0.285, 0.04)Though collaborative tendencies might be common
between Costa Rica and Mexico, help-seeking behavior with the ITS may differ
US and PH – performed very well on each other’s data set (0.146 and 0.228)Reinforces findings in (San Pedro, 2011) wherein a
carelessness detector is generalizable between the two countries
In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries But Help-seeking and Disengaged behavior are two different sets
of behaviors
ConclusionResults did not expose that effective help-
seeking as a whole is very culture-specific (18 out of 25 applications returned positive r)
However, it is not apparent that effective help-seeking is transferrable across countriesBig difference between LOOCV r and cross-
cultural evaluations
Conclusion and RecommendationsThere are pairs of countries wherein effective help-
seeking fail to generalize to each other’s data set.Meaning effective help-seeking from one country may
not necessarily be effective in another
Single effective help-seeking models used by meta-cognitive tutors may be effective in one culture but not in othersFuture meta-cognitive tutors might have to use a
more generalizable modelMay have to switch models, depending on the culture
where the ITS is used