Putting the world to work for ITS:

Aleahmad, Aleven and Kraut

6/27/2008 ITS 2008

Open community authoring of targeted worked example

problems

Current situation in tutoring systems• Development is very laborious

• (e.g. estimates of 200-300 hrs for 1 hr instruction)

• Small groups with much effort per person

• Distribute the development• Open source• Open content

• How to make a “Wikipedia” for ITS?

2

Wikipedia not the right model

3

Towards a collaborative community• Volunteers

submit new material

• Others rate and critique

• Link resources into tutoring

systems or create new ones

• Others make the contribution better

Generate Evaluate

UseImprove

4

Broad research questions

If you make it, will they come? Can the wheat be separated from the

chaff? How to structure and support authoring?

For quality For diversity to engage students

– Contextualization, personalization, and provision of choices can improve student motivation and engagement in learning (Cordova and Lepper, 1996 )

– Personalization improves performance gains and even at start (Anand and Ross, 1987; Ku and Sullivan 2002; López and Sullivan 1992)

5

Overview of the study

Web site where people contribute worked example problems

In registering, indicated their professional status

Tested a mechanism to increase quality and diversity Asked some authors to target to a specific

person Increase their effort? Increase diversity/adaptivity of corpus?

6

Task

• Artifact: Worked example problem– Leads to better and more efficient learning when added

to interactive tutoring (McLaren et al., 2006; Schwonke et al., 2007)

– Instruct and foster self-explanation (Renkl and Atkinson, 2002)

– Customizability – both to the student and the interaction

• Domain: Pythagorean Theorem– Most difficult skill on the Massachusetts Comprehensive

Assessment System curriculum standards (ASSISTment data)

– Diagram drawing difficult to computer generate

7

ProblemStatementSolution steps

+

=Whole worked example8

Work Explanation

Zack and Slater want to build a bike jump. They have two parts of the ramp constructed but they need to know the length of the final piece of the jump. They have two parts of the ramp built, one is 3 ft long and the other is 4 ft long and they are constructed as shown in the diagram. What is the length of the missing section that Zack and Slater still need to construct?

The unknown is the hypotenus which is represented by c in the equation. Therefore I input both a and b into the equation first. Following the equation I square both of these numbers.

These two numbers are added together first because of the parenthesis.

To complete the equation I take the square root of 25 which is five. This problem also demonstrates the common Pythagoras triangle.

9 + 16 = 25

3^2 + 4^2

Square root of 25 is 5 and this is the solution.

Authoring tool9

Open authoring hypotheses

H1: Identifying the good from the bad contributions is easy. We expect that all contributions are good, easily fixed, or easily filtered.

H2: Math teachers submit the best contributions.

10

Student profiles

Goal of realism Varied on social and cognitive attributes 16 profiles

4 Hobbies x 4 Homes 4 realistic skill profiles distributed 2 genders distributed

11

Profile hypotheses

H3: Student profiles lead to tailored contributions.

H4: Student profiles increase the effort of authors.

H5: Student profiles lead to higher quality contributions.

12

Profiles in experimental condition versus generic control condition

Participants and contributions• Participation URL posted on web sites

(educational and otherwise) offering $4-12

• 1427 people registered, of which 570 used the tool to submit 1130 contributions

• After machine filtering, 281 participants were left having submitted 551 contributions

13

Participation Math teachers

Other teachers

Amateurs

Registered 131 170 1126

Contributed also

70 72 428

Passed vetting also

26 35 220

Machine filtered14

Some have just a worthless drawing.

Or nothing at all.

15

Quality ratings

Numerical value

Rating category Definition

0 UselessNo use in teaching and it would be easier to write a new one than improve this one.

1 Easy fixHas some faults, but they are obvious and can be fixed easily, in under 5 minutes.

2 Worthy

Worthy of being given to a student who matches on the difficulty and subject matter. Assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.

3 Excellent

Excellent example to provide to some student. Again, assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.

Human experts rated the machine vetted submissions

Quality rating examples

Excellent statement with poor solution (1124)

Worthy statement with excellent solution (337)

16

Open authoring17

Quality of pool18

Quality by contributor expertise

Statement quality Solution quality

19

Teacher status

Sign. diffs

Mean quality

Std Err

Math teacher

A 1.80 0.12

Other teacher

B 1.54 0.09

Not teacher

B 1.48 0.09

Teacher status

Sign. diffs

Mean quality

Std Err

Math teacher

A B 0.70 0.10

Other teacher

B 0.53 0.08

Not teacher

B 0.76 0.03

Student profiles20

Tailoring to social attributes

AttributeWith

GENERIC(G)

With profiles not

mentioningattribute (N)

With profiles

mentioning attribute

(M)

F-test(G-M)

F-test(N-M)

Female pronoun 5% 4% 16% 9.68* 12.82**

Male pronoun 19% 14% 19% 0.004 1.19

Sports word 9% 9% 24% 18.01** 11.89**

TV word 4% 4% 10% 8.36* 2.63†

Music word 2% 2% 9% 6.92* 8.93**

Home word 14% n/a 20% 3.60* n/a

21

Probabilities of authoring matching an attribute†p<.10 *p<.05 **p<.001

For profile with a home outside town

22

For profile who lives in tall apartment building

23

Tailoring to cognitive attributes

Math skill

shown

Sign.

diffs

Probability of

using 3-4-5

triangle

Std Err

High A 16% 0.05

Medium

A B 26% 0.05

Low B 27% 0.04

GENERIC

A B 21% 0.03

Verbal skill in profileGeneral math skill in profile

24

Correspondence of verbal and math skill levels with the authoring interface

Verbal skill

shown

Sign.

diffs

Mean reading level of

contribution

Std Err

High A 3.78 0.24

Medium

A B 3.56 0.32

Low B 2.93 0.33

GENERIC

B 3.20 0.16

Shakespeare for profile in “top of English class”

25

Effects of profiles

Problem statements in profile condition were 25% longer

No significant difference in time spent (median 5 each minutes on statement and solution)

No main effect of profiles on quality

No interaction with teacher status either

On effort On quality

26

Conclusions27

Recap of Hypotheses

Hypothesis Short Answer

Long Answer

1 Quality control is easy Yes Filtering trivial; rating by experts take less than a minute

2 Math teachers contribute the best worked examples

Partly Amateurs and non-math teachers wrote okay problem statements and amateurs wrote better solutions

3 Profiles lead to tailoring Yes Every aspect of profiles was tailored to

4 Profiles increase effort Inconclusive

A quarter longer problem statement, but no difference in time

5 Profiles lead to higher quality contributions

No No difference in machine filtering or human rated quality

28

• Volunteers submit new material

• Others rate and critique

• Link resources into tutoring

systems or create new ones

• Others make the contribution better

Generate Evaluate

UseImprove

Current and future work29

Current and future work30

• Volunteers submit new material

• Others rate and critique

• Link resources into tutoring

systems or create new ones

• Others make the contribution better

Generate Evaluate

UseImprove

Current and future work31

• Volunteers submit new material

• Others rate and critique

• Link resources into tutoring

systems or create new ones

• Others make the contribution better

Generate Evaluate

UseImprove

Acknowledgements

Thanks to ASSISTment project, Ken Koedinger and Sara Kiesler for data and feedback

Work supported by IES and NSF

It’s going to take a lot of connected work to build a scalable shared ITS for the world Let’s talk more about how http://OpenEducationResearch.org

32

Gratis participants

Still 93 submissions from 92 participants Of these 38 submissions from 21

participants pass machine vetting 41% pass rate of machine vetting

compared to 49% rate in experiment Not significantly different by Fisher's

Exact Test (p=0.16)

33