Jozef Tvarožek and Mária Bieliková

Enhancing Learningwith Off-Task Social Dialogues

EC-TEL 2010, BarcelonaSeptember 30, 2010

Slovak University of Technology in Bratislava

Enhancing Learning with Off-Task Social Dialogues 2

Our Approach to Socially Intelligent Tutor

Enhancing Learning with Off-Task Social Dialogues 3

Learning (1) – Course notes

Enhancing Learning with Off-Task Social Dialogues 4

Learning (2) – Problem solving

Enhancing Learning with Off-Task Social Dialogues 5

Tasks for assessment and practice

Expert’s idea

Task scheme

Task instance

Answer

Answer category

Ability estimateAdaptive selection

Generator

StudentJudge

Estimation

Enhancing Learning with Off-Task Social Dialogues 6

Task scheme specification

A

B

C

A - C

A - B

Solution tree

B - C

Task’s scheme tree

• Parameters, constraints, tree of subtasks and answers• Psychometric IRT parameters, usage indicators

Task scheme:

correct

incorrect

Enhancing Learning with Off-Task Social Dialogues 7

Task instance

Parameters’ specification

Instantiated parameters

Scheme tree Instance tree

Pruned backtracking

Combine

Instance generation

Enhancing Learning with Off-Task Social Dialogues 8

Updating user’s profile

• Off-task dialogues– Qs/As scripted to perform actions• Extracting user’s preferences & behaviors• Extracting event attributes• Recommending events to attend• Negotiating events with others

– Relationship maintenance

Enhancing Learning with Off-Task Social Dialogues 9

Extracting interests

Tutor: now, look around and solve exercises, ok? see you around!

Student ack / Turn initiative

Tutor: interesting, I for example like to read books, swim, play volleyball and soccer

≥ 40 chars< 40 chars

Extract features (e.g. to draw, watch TV, friends)

Tutor: Hello Kate, how are you? I'm here to make you feel comfortable, so that you learn much... :-)

Tutor: ok, write me about yourself, what you like, and all... I can then prepare exercises that you will like ... ;)

Student greeting

Tuto

r: te

ll m

e m

ore,

pls

.

Enhancing Learning with Off-Task Social Dialogues 10

Sample conversation

Joe:Hello Kate, how are you?I’m here to make you feel comfortable, so that you learn much… :-)

Kate: thnx

Joe:ok, write me about yourself, what you like, and all…I can then prepare some exercises that you will definitely like :P

Kate: like to draw sleep watch TV …

Joe: write a bit more please …

Kate: like to go out go out with my dog

Joe:interesting, I for example like to read books, from sports volleyball, swimming, and also some soccer

Kate: okay

Joe: now, look around this environment and solve some of the exercises, ok?see you around!

Enhancing Learning with Off-Task Social Dialogues 11

Real-life adaptation of tasks

Parameters’ specification

Instantiated parameters

Scheme tree Instance tree

Pruned backtracking

Combine

Instance generation, guided by student’s hobbies

Semantic similarity with student’s

favorite concepts

Enhancing Learning with Off-Task Social Dialogues 12

Evaluation study

• Middle school mathematics• 18 parametric algebra tasks• Tutoring friend– Extract hobbies

• Students did participate in a pilot previously– Familiar with the environment

Enhancing Learning with Off-Task Social Dialogues 13

Is it better than paper&pencil?

• 32 students– Control group = traditional classroom– Experimental group = tutor

– Learning gain: 1.2% vs 10.3%

pre-test post-test gain t-test

mean st.dev mean st.dev mean st.dev t stat p-value

Control group 0.697 0.230 0.709 0.211 0.012 0.258 -0.18 0.428

Exp. group 0.434 0.253 0.538 0.269 0.103 0.172 -2.40 0.015

Enhancing Learning with Off-Task Social Dialogues 14

Are they willing to do it?

• 16 students• Detect student interests in the initial welcome

dialogue: to draw, sleep, watch TV, go out, go out with dog

• Mean word count 11.6 (st.dev 8.7)• Mean feature count 1.56 (st.dev 1.7)• 44% IGNORED the tutor– Others: mfc 2.78 (st.dev 1.39)

Enhancing Learning with Off-Task Social Dialogues 15

Motivating students?

• Those that did engage with the tutor– Less problems attempted, higher success rate.

Control Experimentalmean st.dev Mean st.dev

Number of tasks attempted 7.71 3.86 7.00 1.80

Number of tasks solved correctly 2.85 1.46 4.00 2.45

Questions (response scale: 1=worst, 5=best)

1. How much did you learn in the tutor? 2.29 1.25 3.33 0.71

2. How much did the tutor help you on the post-test?

2.29 1.11 3.33 1.12

3. How much would you like to use the tutor again?

3.14 1.07 4.33 1.12

4. How did you like the tutor? 2.86 1.07 4.22 0.97

Enhancing Learning with Off-Task Social Dialogues 16

Motivating students? (contd.)

• Is the tutoring friend any good?– We don’t know.

– Learning gain: 3.7% vs. 12.3%– We can filter students that are

engaged, and do well.

pre-test post-test gain

mean st.dev mean st.dev mean st.dev

Engaged 0.429 0.245 0.465 0.283 0.037 0.283

Not engaged 0.439 0.273 0.562 0.284 0.123 0.192

Enhancing Learning with Off-Task Social Dialogues 17

Summing up

• Those who engage in the social off-task dialog with the tutor solve problems better :)

• Tutors that are “friends” with students can produce higher learning gains.

• Socially intelligent tutor – tutoring friend:– gets to know you better,– guides you to what you need.

Jozef Tvarožek and Mária Bieliková

Enhancing Learningwith Off-Task Social Dialogues

EC-TEL 2010, BarcelonaSeptember 30, 2010

Slovak University of Technology in Bratislava