department of computer science ivon arroyo university of massachusetts amherst lessons learned in...
Post on 05-Jan-2016
213 Views
Preview:
TRANSCRIPT
Department of Computer Science
Ivon ArroyoUniversity of Massachusetts Amherst
Lessons Learned in Teaching Mathematics with Adaptive
Tutoring Software
To Erica
2
Wayang Outpost --Math Tutoring SystemGrades 7,8,9,10 and community colleges http://Wayangoutpost.com
3
MCASpassing%Wayang
MCASpassing%
No Wayang
77% 60% **
34% 24% *
92% 76% *
WayangPosttest
ControlNo Wayang
76% 67% **
d=0.25
d=0.24
d=0.52
Empirical Learning Results since 2003After short exposure (3-4 hours)
Expanding to 2000 students in 2011
4
Wayang Outpost --Math Tutoring SystemStandardized-test math problems with multimedia help
More Help
http://Wayangoutpost.com
ModalityAnimationContiguity
5
What have we learned about how to teach math with advanced technologies?
Many things….
Some are supported by experimental evidence
Some are conjectures and anecdotes…
6
What have we learned about teaching math?
Showing Progress
Adaptive Problem Selection
Affect
Training Math Fluency
Offering Help
7
Lesson learned 1
Adaptive Math Tutoring that maintains students within a “zone of proximal development”
improves learning.
88
What kind of adaptivity?Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758
Can we understand when we are outside of the ZPD?
Little Effort
Too much effort
Frustrated
Fatigued
99
E(Ii)
IL IH
E(Hi)
HL HH
E(Ti)
TLTH
0 1 2 3 4 0 1 2 3 4 5 6 7
Incorrect Attempts Hints Time (each bar=5seconds)
Attempts < E(Ii) — IL Hints > E(Hi) + HH Time < E(Ti) — TL
Odd behavior: too much effort, or too little effort
Few Inc. Attempts Lots of Hints Little Time< > <
Learning what is high and low effortIn any problem pi i=1, .., N N=Total problems in system
10
Scenarios outside of the ZPD
1111
Where the ZPD is
dHIGH
dLOW
Little Effort
Too much effort
Frustrated
Fatigued
Disengaged
Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758
12
How to increase/decrease problem difficulty?
Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining.
Pittsburgh, PA.
13
Does this adaptivity improve learning?Randomized Controlled Experiment (N=56) Spring 2004
ANCOVA for Posttest Score F(55,1)=8.4, p=.006
Raw percent Correct (Pre and Posttest) Accuracy over attempted problems
Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining.
Pittsburgh, PA.
14
Lesson learned 1
Adaptive Math Tutoring that attempts to maintain students within a “zone of proximal development” improves learning.
Being adaptive over smaller “chunks” of similar problems (instead of the full set of problems)
yields higher learningBeing “gentle” at increasing difficulty yields
higher learning
15
Lesson learned 2
Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.
16
Math Facts Retrieval (fluency) training
Royer, J. M., & Tronsky, L. N. (1998). Addition practice with math disabled students improves subtraction and multiplication performance. In T. E. Scruggs and M. A. Mastropieri (Eds.), Advances in Learning and Behavioral Disabilities (Vol 12). Greenwich, Conn.: JAI Press, Inc.
1717
True Means and SD for HARD items of the standardized pretest and posttest
ANCOVA for Hard Posttest with Hard Pretest as covariate; MFR,Wayang fixed factors:Wayang F(222,1)=6.8, p=.01; WayangxMFR F(222,1)=6.8, p=.009
Post-Hoc Contrasts: Wayang > no-Wayang? Yes. Wayang-MFR > Wayang-NoMFR? Yes.
Results on Standard Math Test (Hard Items)
18
But why? Problem solving takes place in a cognitive system constrained by
a limited capacity of working memory
StrategyBasic Math
Working Memory Capacity When Solving a Math Problem
Doing Math is like speaking a language. If you are fluent, you will concentrate better on the message.
Basic Math
Math FluencyMath Fluency helps predict performance at state-wide tests“Basic Math” could be anything… such as solving easy equations…
Royer&Tronsky (1998)Royer et al. (1999)
19
Lesson learned 2
Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.
20
Lesson learned 3
Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems
and increases learning.
2121
5 problems earlier Last 5 problems
Similar to open learner models, but focus on progress
Progress Monitoring InterventionsSelf-monitoring feedback Self-referenced-feedback (McColskey and Leary (1985))
2222
5 problems earlier Last 5 problems
Progress Monitoring InterventionsSelf-monitoring feedback Self-referenced-feedback (McColskey and Leary (1985))
2323
Mathematics Learning Results
+0%
+7%
Means and Standard Deviations Percentages
+16%
+13%
ANCOVA
Dependent Learning (Posttest-Pretest score)GroupCovariate Pretest score
F=4.23, p=.043
Effect size: Cohen’s d=0.4
24
Changes in time per problemMedians and Quartiles for time spent in the problem (in pairs of subsequent problems)
529303 529303N =
Seconds spent in a problem
MotivationalControl
Media
n/q
uart
iles
for
tim
e s
pent
per
pro
ble
m70
60
50
40
30
20
10
0
Before Intervention
After Intervention
Tutor-InterventionTutor-Control
25
Lesson learned 3
Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems
and increases learning.
26
Lesson learned 4
Emotions/Attitudes/Affect as a more important long-term outcome than learning.
In general, students are really bored about mathematics.
Also, there are important group differences in students emotions, before tutoring.
27
Who needs more affective supportLow achieving students (95% of students with IEPs)
Table 1 . Affective self-reports of high-achieving vs. low-achieving students prior to Tutoring
Af fective Criterion Means, standard deviations and between-
subjects test Lo w-achie ving: N=64; High-achie ving: N=43
Self-concept of math abili ty (in comparison to other stud ents,
other subjects, 3 items)
Lo w-achie ving: M=3.2 SD=1.1 High-achie ving: M=4.1 SD=1.0
***F(106,1)=18.2, p=.000
How confident do you feel when solvi ng math prob lems?
Lo w-achie ving: M=3.1 SD=1.3 High-achie ving: M=4.0 SD=1.3
***F(105,1)=11.5, p=.001
How frus trating is it to solve math problems?
Lo w-achie ving: M=3.6 SD=1.2 High-achie ving: M=3.0 SD=1.1
** F(106,1)=7.6, p=.007
How exciting is it to solve math prob lems?
Lo w-achie ving: N=64 , M=2.2 SD=1.2 High-achie ving: N=43, M=2.7 SD=1.4
*F(106,1)=3.64, p=0.05
28Figure 1: Results for a pre-tutor survey in two public schools: Girls
develop negative feelings for mathematics, including decreased confidence (left) and increased frustration (right), between middle and
high school.
Who needs more affective supportHigh School Girls
29
Lesson learned 4
Emotions/Attitudes/Affect as a more important long-term outcome than learning.
In general, students are really bored about mathematics.
Also, there are important group differences in students emotions, before tutoring.
30
Lessons learned 5
We can understand and TRACE students at the affective level, understanding their emotions.
How? from very recent behaviors, and with the help of physiological sensors.
3131
Students Self-Report EmotionsEvery 5 minutes, students would report emotions
32
Affective Tracing
Anxiety BoredomFrustration
Predicting Emotions in Real-Time
Linear Models to Predict Emotions from last Problem SeenModels Created using Stepwise Regression
# Hints Seen
Solved? 1st Attempt
# Incorrectattempts
CharacterPresent?
Seconds to1st Attempt
Time in Tutor
Seconds To Solve
Tutor Context Variables (for the last problem)
R2=0.3 R2=0.15 R2=0.18R2=0.19
Enjoyment
Accuracy of a YES/NO prediction of each emotion, compared to TRUE self-report
86% 88% 78% 83%
Possible that looking at longer episodes of recent history will achieve as good accuracy as sensors.
SitForwardStdev
“Concen-trating”
SitForwardMean
R2=0.38 R2=0.31R2
=0.40
R2=0.44
“Interest”Min
MaxPressure
33
Lessons learned 5
We can understand and TRACE students at the affective level, understanding their emotions.
How? from very recent behaviors, and with the help of physiological sensors.
34
Lessons learned 6
Affective Characters that talk about the importance of Effort and Perseverance improve affect
towards math for all, particularly for girls and low achieving students.
However, they don’t impact learning.
35
Animated Pedagogical Agents
“Cognitive”Pedagogical
Agents
Cognitive Outcomes(Retention, Transfer)
Affective Outcomes(Motivation, Attitudes, Emotions)
AffectivePedagogical
Agents?
36
Human-Like Affective Learning CompanionsAffective Experts, cognitive peers
Train the idea of “Malleability of Intelligence”Dweck, C.S., (1999) Self-Theories: Their role in motivation, personality and development
37
Human-Like Affective Learning CompanionsAffective Experts, cognitive peers
Quick-guess incorrect
Correct No EffortPraise Effort and Time on Hints
Low Effort High Effort
Incorrect “We kind of rushed to answer that one. Shall we ask the computer for help? I am sure we will get
it if we take the time to solve the problem.”
“These are the hard questions that I like. There is an opportunity to learn. Let’s click on the help button.”
Correct “That was good, however, I prefer harder questions so that we learn from the help that the
computer gives, even if we get them wrong.”
“Hey, congratulations! Your effort paid off, you got it right!”
38
Summary of ResultsAnalysis of Covariance
Affective Learning Companions are good for all
39Frustrated Pretest Frustrated Within
TutorFrustrated Posttest
1.5
2
2.5
3
3.5
4
4.5
5
No Learning CompanionLearning Companion
How
FR
US
TR
ATE
D d
o y
ou
feel w
hen
solv
ing
m
ath
pro
ble
ms?
Impact of Affective LCs for all
Less frustration reported within the tutor with Jane. **F(213,2)=6.1,p=.003
More Frustrated
Less frustrated
NeutralFrustration
Level
40
Impact of Affective LCs for all
Less boredom for math at posttest time in LC condition.
For N~95 students, comparing LCs vs. no-LCs
Interested Pretest Interested Within Tutor Interested Posttest0.6
1.1
1.6
2.1
2.6
3.1
3.6
4.1
No Learning Companion Learning Companion
How
IN
TE
RE
STE
D a
re y
ou
wh
en
solv
ing
m
ath
pro
ble
ms?
+F(94,1)=3.4,p=.07
More Interested
More Bored
NeutralInterestLevel
41
Impact of Affective LCs for LOW ACHIEVING
More CONFIDENCE for math at posttest time FOR LOW ACHIEVING.
For N~95 students, comparing LCs vs. no-LCs
42
Lessons learned 6
Affective Characters that talk about the importance of Effort and Perseverance improve affect towards math for all and for low achieving
students.
However, they don’t impact learning.
43
Lessons learned 7
There are important gender differences that suggest girls have more productive use of
Wayang
Keep in mind that you might be designing for a subset of the population
44Affective Learning Companions are good for all
Benefit for all, but effect is stronger when considering Girls alone
Summary of Results charactersAnalysis of Covariance
45See dotted line for increased frustration without companions
Frustration Level
Impact of Affective LCs on GIRLS
More Frustrated
Less frustrated
Reduced FrustrationWith “Jane”
NeutralFrustration
Level
46
Frustrated Pre-Tutor
Frustrated Within Tutor
Frustrated Post-Tutor
1
3.5
6
Males without Learning CompanionMales with JakeMales with Jane
Mean
Fru
stra
tion
an
d S
tdev (
6=
very
fru
stra
ted
)
Frustration Level
Impact of Affective LCs on BOYS
More Frustrated
Less frustrated
NeutralFrustration
Level
47Affective Learning Companions are good for all
Overall benefit, but the effect is stronger for Girls alone.Girls are the ONLY ones benefitting, and boys are clearly not.
Summary of Results about charactersAnalysis of Covariance
48
Perceptions of Wayang Outpost
48
Liked it? Did you Learn? Was Wayang concerned about your learning? Was it helpful?
Means and S.E. for overall Perception of Wayang Outpost
Positive perception
Negativeperception
Girls report a betterlearning experience
With ALCs.
Boys report a better
experienceWithout ALCs.
Males Females Males Females
No LearningCompanion
Learning Companion
NeutralPerception
(neither positive nor negative)
49
Gender differences in Accepting/Rejecting help
Help Offered
Help AcceptedHelp Rejected
50
Gender differences in attitudes and behaviors in Wayang
51
Lessons learned 7
There are important gender differences that suggest girls have more productive use of
Wayang
Keep in mind that you might be designing for a subset of the population
52
Lessons learned
1) Adaptively Maintaining students within a “zone of proximal development” improves learning.
2) Training Math Fluency enhances mathematics learning in combination with adaptive tutoring.
3) Showing students their historical improvement at math problem solving improves engagement in subsequent
problems and increases learning.4) Students are really bored about mathematics. Also, there are
important group differences in emotions, before tutoring5) We can understand and TRACE students at the affective level,
understanding their emotions.6) Affective Characters improve affect. However, they don’t
impact learning.7) Keep in mind that you might be designing for a subset of the
population, such as girls and low achieving students.
53
Improving learning by looking at cognition alone: • Reduce working memory load• Facilitate transfer by keeping similar problems together
Keep concepts in working memory
Improving engagement (and affect?) is related to:Pacing and fatigueSupporting meta-cognition, goal setting, reflection.Looking at individual groups of students who need it most
Improving affect, emotions and long term attitudes:Being Positive, encouraging feedbackReflecting about myths and training attributions for failure
Concluding… My conjecturesHow to improve learning, affect, engagement
Department of Computer Science
Ivon ArroyoUniversity of Massachusetts Amherst
Lessons Learned in Teaching Mathematics with Adaptive
Tutoring Software
To Erica
55
56
Lessons learned
If being “gently adaptive” is better…
Shall we be adaptive over the whole pool of problems, or over smaller chunks (topics,skills)?
57
Lesson learned 3
Chunking problems (grouping problems of similar skills together) improves learning.
5858
Results on Standard Math Test (Easy Items)
ANCOVA for Easy Posttest with Easy Pretest as covariate; MFR,Wayang fixed factors:Wayang F(222,1)=10.6, p=.001; WayangxMFR F(222,1)=5.1, p=.025
Post-hoc Contrasts: Wayang > no-Wayang? Yes. Wayang-MFR > Wayang-NoMFR? No.
True Means and SD for EASY items of the standardized pretest and posttest
59
Lesson learned 1.a
How fast should we increase/decrease problem difficulty?
60
Varying the challenge in adaptive problem selection
XX
Challenge Wayang
GentleWayang
61
“Gentle Adaptive” Wayang also offers Help
Unpublished
62
Scaffolding and Help Offering
Pretest % correct (11 questions) Posttest %correct (11 questions)15%
25%
35%
45%
55%
65%
75%
Challenge Problem Selector
Gentle Problem SelectorP
rete
st
and P
ostt
est
Sco
res
Unpublished
63
Chunking similar problems together or Not
Pretest % correct (11 questions) Posttest %correct (11 questions)0%
10%
20%
30%
40%
50%
60%
70%
No chunking ChunkingP
rete
st
and P
ostt
est
Sco
res
Chunking facilitates transfer?
top related