Department of Computer Science

Ivon ArroyoUniversity of Massachusetts Amherst

Lessons Learned in Teaching Mathematics with Adaptive

Tutoring Software

To Erica

2

Wayang Outpost --Math Tutoring SystemGrades 7,8,9,10 and community colleges http://Wayangoutpost.com

3

MCASpassing%WayangMCASpassing%No Wayang

77% 60% **

34% 24% *

92% 76% *

WayangPosttest ControlNo Wayang76% 67% **

d=0.25

d=0.24

d=0.52

Empirical Learning Results since 2003After short exposure (3-4 hours)

Expanding to 2000 students in 2011

4

Wayang Outpost --Math Tutoring SystemStandardized-test math problems with multimedia help

More Help

http://Wayangoutpost.com

ModalityAnimationContiguity

5

What have we learned about how to teach math with advanced technologies?

Many things….

Some are supported by experimental evidence

Some are conjectures and anecdotes…

6

What have we learned about teaching math?

Showing Progress

Adaptive Problem Selection

Affect

Training Math Fluency

Offering Help

7

Lesson learned 1

Adaptive Math Tutoring that maintains students within a “zone of proximal development”

improves learning.

88

What kind of adaptivity?Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758

Can we understand when we are outside of the ZPD?

Little Effort

Too much effort

Frustrated

Fatigued

99

E(Ii)

IL IH

E(Hi)

HL HH

E(Ti)

TLTH

0 1 2 3 4 0 1 2 3 4 5 6 7

Incorrect Attempts Hints Time (each bar=5seconds)

Attempts < E(Ii) — IL Hints > E(Hi) + HH Time < E(Ti) — TL

Odd behavior: too much effort, or too little effort

Few Inc. Attempts Lots of Hints Little Time< > <

Learning what is high and low effortIn any problem pi i=1, .., N N=Total problems in system

10

Scenarios outside of the ZPD

1111

Where the ZPD is

HIGH

LOW

Little Effort

Too much effort

Frustrated

Fatigued

Disengaged

Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758

12

How to increase/decrease problem difficulty?

Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining.

Pittsburgh, PA.

13

Does this adaptivity improve learning?Randomized Controlled Experiment (N=56) Spring 2004

ANCOVA for Posttest Score F(55,1)=8.4, p=.006

Raw percent Correct (Pre and Posttest) Accuracy over attempted problems

Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining.

Pittsburgh, PA.

14

Lesson learned 1

Adaptive Math Tutoring that attempts to maintain students within a “zone of proximal development” improves learning.

Being adaptive over smaller “chunks” of similar problems (instead of the full set of problems)

yields higher learningBeing “gentle” at increasing difficulty yields

higher learning

15

Lesson learned 2

Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.

16

Math Facts Retrieval (fluency) training

Royer, J. M., & Tronsky, L. N. (1998). Addition practice with math disabled students improves subtraction and multiplication performance. In T. E. Scruggs and M. A. Mastropieri (Eds.), Advances in Learning and Behavioral Disabilities (Vol 12). Greenwich, Conn.: JAI Press, Inc.

1717

True Means and SD for HARD items of the standardized pretest and posttest

ANCOVA for Hard Posttest with Hard Pretest as covariate; MFR,Wayang fixed factors:Wayang F(222,1)=6.8, p=.01; WayangxMFR F(222,1)=6.8, p=.009

Post-Hoc Contrasts: Wayang > no-Wayang? Yes. Wayang-MFR > Wayang-NoMFR? Yes.

Results on Standard Math Test (Hard Items)

18

But why? Problem solving takes place in a cognitive system constrained by

a limited capacity of working memory

StrategyBasic Math

Working Memory Capacity When Solving a Math Problem

Doing Math is like speaking a language. If you are fluent, you will concentrate better on the message.

Basic Math

Math FluencyMath Fluency helps predict performance at state-wide tests“Basic Math” could be anything… such as solving easy equations…

Royer&Tronsky (1998)Royer et al. (1999)

19

Lesson learned 2

Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.

20

Lesson learned 3

Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems

and increases learning.

2121

5 problems earlier Last 5 problems

Similar to open learner models, but focus on progress

Progress Monitoring InterventionsSelf-monitoring feedback Self-referenced-feedback (McColskey and Leary (1985))

2222

5 problems earlier Last 5 problems

Progress Monitoring InterventionsSelf-monitoring feedback Self-referenced-feedback (McColskey and Leary (1985))

2323

Mathematics Learning Results

+0%+7%

Means and Standard Deviations Percentages

+16%

+13%

ANCOVADependent Learning (Posttest-Pretest score)GroupCovariate Pretest score

F=4.23, p=.043

Effect size: Cohen’s d=0.4

24

Changes in time per problemMedians and Quartiles for time spent in the problem (in pairs of subsequent problems)

529303 529303N =

Seconds spent in a problem

MotivationalControl

Med

ian/

quar

tiles

for t

i me

spen

t per

pro

blem

70

60

50

40

30

20

10

0Before InterventionAfter Intervention

Tutor-InterventionTutor-Control

25

Lesson learned 3

Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems

and increases learning.

26

Lesson learned 4

Emotions/Attitudes/Affect as a more important long-term outcome than learning.

In general, students are really bored about mathematics.

Also, there are important group differences in students emotions, before tutoring.

27

Who needs more affective supportLow achieving students (95% of students with IEPs)

Table 1 . Affective self-reports of high-achieving vs. low-achieving students prior to Tutoring

Af fective Criterion Means, standard deviations and between-

subjects test Lo w-achie ving: N=64; High-achie ving: N=43

Self-concept of math abili ty (in comparison to other stud ents,

other subjects, 3 items)

Lo w-achie ving: M=3.2 SD=1.1 High-achie ving: M=4.1 SD=1.0

***F(106,1)=18.2, p=.000

How confident do you feel when solvi ng math prob lems?

Lo w-achie ving: M=3.1 SD=1.3 High-achie ving: M=4.0 SD=1.3

***F(105,1)=11.5, p=.001

How frus trating is it to solve math problems?

Lo w-achie ving: M=3.6 SD=1.2 High-achie ving: M=3.0 SD=1.1

** F(106,1)=7.6, p=.007

How exciting is it to solve math prob lems?

Lo w-achie ving: N=64 , M=2.2 SD=1.2 High-achie ving: N=43, M=2.7 SD=1.4

*F(106,1)=3.64, p=0.05

28Figure 1: Results for a pre-tutor survey in two public schools: Girls

develop negative feelings for mathematics, including decreased confidence (left) and increased frustration (right), between middle and

high school.

Who needs more affective supportHigh School Girls

29

Lesson learned 4

Emotions/Attitudes/Affect as a more important long-term outcome than learning.

In general, students are really bored about mathematics.

Also, there are important group differences in students emotions, before tutoring.

30

Lessons learned 5

We can understand and TRACE students at the affective level, understanding their emotions.

How? from very recent behaviors, and with the help of physiological sensors.

3131

Students Self-Report EmotionsEvery 5 minutes, students would report emotions

32

Affective Tracing

Anxiety BoredomFrustration

Predicting Emotions in Real-Time

Linear Models to Predict Emotions from last Problem SeenModels Created using Stepwise Regression

# Hints Seen

Solved? 1st Attempt

# Incorrectattempts

CharacterPresent?

Seconds to1st Attempt

Time in Tutor

Seconds To Solve

Tutor Context Variables (for the last problem)

R2=0.3 R2=0.15 R2=0.18R2=0.19

Enjoyment

Accuracy of a YES/NO prediction of each emotion, compared to TRUE self-report

86% 88% 78% 83%

Possible that looking at longer episodes of recent history will achieve as good accuracy as sensors.

SitForwardStdev

“Concen-trating”

SitForwardMean

R2=0.38 R2=0.31R2

=0.40

R2=0.44

“Interest”Min

MaxPressure

33

Lessons learned 5

We can understand and TRACE students at the affective level, understanding their emotions.

How? from very recent behaviors, and with the help of physiological sensors.

34

Lessons learned 6

Affective Characters that talk about the importance of Effort and Perseverance improve affect

towards math for all, particularly for girls and low achieving students.

However, they don’t impact learning.

35

Animated Pedagogical Agents

“Cognitive”Pedagogical

Agents

Cognitive Outcomes(Retention, Transfer)

Affective Outcomes(Motivation, Attitudes, Emotions)

AffectivePedagogical

Agents?

36

Human-Like Affective Learning CompanionsAffective Experts, cognitive peers

Train the idea of “Malleability of Intelligence”Dweck, C.S., (1999) Self-Theories: Their role in motivation, personality and development

37

Human-Like Affective Learning CompanionsAffective Experts, cognitive peers

Quick-guess incorrect

Correct No EffortPraise Effort and Time on Hints

Low Effort High Effort

Incorrect “We kind of rushed to answer that one. Shall we ask the computer for help? I am sure we will get

it if we take the time to solve the problem.”

“These are the hard questions that I like. There is an opportunity to learn. Let’s click on the help button.”

Correct “That was good, however, I prefer harder questions so that we learn from the help that the

computer gives, even if we get them wrong.”

“Hey, congratulations! Your effort paid off, you got it right!”

38

Summary of ResultsAnalysis of Covariance

Affective Learning Companions are good for all

39Frustrated Pretest Frustrated Within

TutorFrustrated Posttest

1.5

2

2.5

3

3.5

4

4.5

5

No Learning CompanionLearning Companion

How

FRUS

TRAT

ED d

o yo

u fe

el w

hen

solv

ing

mat

h pr

oble

ms?

Impact of Affective LCs for allLess frustration reported within the tutor with Jane. **F(213,2)=6.1,p=.003

More Frustrated

Less frustrated

NeutralFrustration

Level

40

Impact of Affective LCs for all

Less boredom for math at posttest time in LC condition.For N~95 students, comparing LCs vs. no-LCs

Interested Pretest Interested Within Tutor Interested Posttest0.6

1.1

1.6

2.1

2.6

3.1

3.6

4.1

No Learning Companion Learning Companion

How

INTE

REST

ED a

re y

ou w

hen

solv

ing

mat

h pr

oble

ms?

+F(94,1)=3.4,p=.07

More Interested

More Bored

NeutralInterestLevel

41

Impact of Affective LCs for LOW ACHIEVING

More CONFIDENCE for math at posttest time FOR LOW ACHIEVING.For N~95 students, comparing LCs vs. no-LCs

42

Lessons learned 6

Affective Characters that talk about the importance of Effort and Perseverance improve affect towards math for all and for low achieving

students.

However, they don’t impact learning.

43

Lessons learned 7

There are important gender differences that suggest girls have more productive use of

Wayang

Keep in mind that you might be designing for a subset of the population

44Affective Learning Companions are good for allBenefit for all, but effect is stronger when considering Girls alone

Summary of Results charactersAnalysis of Covariance

45See dotted line for increased frustration without companions

Frustration Level

Impact of Affective LCs on GIRLS

More Frustrated

Less frustrated

Reduced FrustrationWith “Jane”

NeutralFrustration

Level

46

Frustrated Pre-Tutor

Frustrated Within Tutor

Frustrated Post-Tutor

1

3.5

6

Males without Learning CompanionMales with JakeMales with Jane

Mea

n Fr

ustra

tion

and

Stde

v (6

=ve

ry fr

ustra

ted)

Frustration Level

Impact of Affective LCs on BOYS

More Frustrated

Less frustrated

NeutralFrustration

Level

47Affective Learning Companions are good for all

Overall benefit, but the effect is stronger for Girls alone.Girls are the ONLY ones benefitting, and boys are clearly not.

Summary of Results about charactersAnalysis of Covariance

48

Perceptions of Wayang Outpost

48

Liked it? Did you Learn? Was Wayang concerned about your learning? Was it helpful?

Means and S.E. for overall Perception of Wayang Outpost

Positive perception

Negativeperception

Girls report a betterlearning experience

With ALCs.

Boys report a better

experienceWithout ALCs.

Males Females Males Females

No LearningCompanion

Learning Companion

NeutralPerception

(neither positive nor negative)

49

Gender differences in Accepting/Rejecting help

Help Offered

Help AcceptedHelp Rejected

50

Gender differences in attitudes and behaviors in Wayang

51

Lessons learned 7

There are important gender differences that suggest girls have more productive use of

Wayang

Keep in mind that you might be designing for a subset of the population

52

Lessons learned1) Adaptively Maintaining students within a “zone of proximal

development” improves learning.2) Training Math Fluency enhances mathematics learning in

combination with adaptive tutoring.3) Showing students their historical improvement at math

problem solving improves engagement in subsequent problems and increases learning.

4) Students are really bored about mathematics. Also, there are important group differences in emotions, before tutoring

5) We can understand and TRACE students at the affective level, understanding their emotions.

6) Affective Characters improve affect. However, they don’t impact learning.

7) Keep in mind that you might be designing for a subset of the population, such as girls and low achieving students.

53

Improving learning by looking at cognition alone: • Reduce working memory load• Facilitate transfer by keeping similar problems together

Keep concepts in working memory

Improving engagement (and affect?) is related to:Pacing and fatigueSupporting meta-cognition, goal setting, reflection.Looking at individual groups of students who need it most

Improving affect, emotions and long term attitudes:Being Positive, encouraging feedbackReflecting about myths and training attributions for failure

Concluding… My conjecturesHow to improve learning, affect, engagement

Department of Computer Science

Ivon ArroyoUniversity of Massachusetts Amherst

Lessons Learned in Teaching Mathematics with Adaptive

Tutoring Software

To Erica

55

56

Lessons learned

If being “gently adaptive” is better…

Shall we be adaptive over the whole pool of problems, or over smaller chunks (topics,skills)?

57

Lesson learned 3

Chunking problems (grouping problems of similar skills together) improves learning.

5858

Results on Standard Math Test (Easy Items)

ANCOVA for Easy Posttest with Easy Pretest as covariate; MFR,Wayang fixed factors:Wayang F(222,1)=10.6, p=.001; WayangxMFR F(222,1)=5.1, p=.025

Post-hoc Contrasts: Wayang > no-Wayang? Yes. Wayang-MFR > Wayang-NoMFR? No.

True Means and SD for EASY items of the standardized pretest and posttest

59

Lesson learned 1.a

How fast should we increase/decrease problem difficulty?

60

Varying the challenge in adaptive problem selection

XX

Challenge WayangGentle

Wayang

61

“Gentle Adaptive” Wayang also offers Help

Unpublished

62

Scaffolding and Help Offering

Pretest % correct (11 questions) Posttest %correct (11 questions)15%

25%

35%

45%

55%

65%

75%

Challenge Problem SelectorGentle Problem Selector

Pret

est

and

Post

test

Sco

res

Unpublished

63

Chunking similar problems together or Not

Pretest % correct (11 questions) Posttest %correct (11 questions)0%

10%

20%

30%

40%

50%

60%

70%

No chunking ChunkingPr

etes

t an

d Po

stte

st S

core

s

Chunking facilitates transfer?