Computer Supported Computer Supported

Collaborative LearningCollaborative Learning

Language Technologies InstituteCarnegie Mellon University

Wednesday, March 18, 2009: Speech and NLP for Educational Applications Wednesday, March 18, 2009: Speech and NLP for Educational Applications

22/44/44

Research GroupResearch Group

Rohit KumarRohit Kumar

33/44/44

Our work: QuestionsOur work: Questions Conversational Agents Conversational Agents (Among other things)(Among other things)

To support human users at various tasksTo support human users at various tasks What kind of tasks?What kind of tasks?

Supporting Learning tasksSupporting Learning tasks What kind of support? (Agents ofcourse, but…)What kind of support? (Agents ofcourse, but…)

What role?What role? ManifestationManifestation How do we make sure the support is getting through?How do we make sure the support is getting through?

What kind of users?What kind of users? Individuals / Pairs / Groups ?Individuals / Pairs / Groups ? Students ? Teachers/facilitators?Students ? Teachers/facilitators?

What environment?What environment? How do we build this support?How do we build this support? How do we evaluate if the support helps?How do we evaluate if the support helps?

44/44/44

Our focus today:Our focus today:

Rohit Kumar, Carolyn P. Rosé, Yi-Chia Wang,Rohit Kumar, Carolyn P. Rosé, Yi-Chia Wang,

Mahesh Joshi, Allen RobinsonMahesh Joshi, Allen Robinson

Tutorial Dialogue as Adaptive Tutorial Dialogue as Adaptive Collaborative Learning Collaborative Learning SupportSupportArtificial Intelligence in Education 2007Artificial Intelligence in Education 2007

Particularly interesting:Particularly interesting:

Borderlining the transition of the CycleTalk project Borderlining the transition of the CycleTalk project from Tutorial Dialog to CSCLfrom Tutorial Dialog to CSCL

55/44/44

CycleTalkCycleTalk Cycle PadCycle Pad

(Forbus et. al. 1999)(Forbus et. al. 1999) Thermodynamic cycles Thermodynamic cycles

simulation environmentsimulation environment Designed to engage Designed to engage

students in engineering students in engineering designdesign

Exploratory learningExploratory learning

Cycle Talk Goals:Cycle Talk Goals: Support engineering Support engineering

students learning the students learning the principles involved in principles involved in designing thermodynamic designing thermodynamic cyclescycles

Study the benefits of Study the benefits of tutorial dialogue in an tutorial dialogue in an exploratory learning contextexploratory learning context

66/44/44

CycleTalk: HistoryCycleTalk: History Rosé et. al., CycleTalk: Towards a Dialogue Agent that Rosé et. al., CycleTalk: Towards a Dialogue Agent that

Guides Design with an Articulate Simulator, 2004Guides Design with an Articulate Simulator, 2004 Cognitive Task AnalysisCognitive Task Analysis

Observation: Cycle Pad’s significant pedagogical potential Observation: Cycle Pad’s significant pedagogical potential tends to be underutilized when students do not receive tends to be underutilized when students do not receive tutorial guidancetutorial guidance

Buildcycle

Incorporate conceptual

understanding

Explore relationships

between parameters

Generate plan to improve cycle

Assume component parameters

Investigate variable

dependencies

Compare multiple cycle improvements

Compare cycleto alternatives

77/44/44

CycleTalk: HistoryCycleTalk: History Rosé et. al., A First Evaluation of the Instructional Value of Rosé et. al., A First Evaluation of the Instructional Value of

Negotiable Problem Solving Goals on the Exploratory Negotiable Problem Solving Goals on the Exploratory Learning Continuum, 2005Learning Continuum, 2005

3 conditions3 conditions NPSG: Human Tutoring + written material (script)NPSG: Human Tutoring + written material (script) PS: Example tracing tutors (Aleven et. al.) + scriptPS: Example tracing tutors (Aleven et. al.) + script S: Script onlyS: Script only

Overall order: PS < S < NPSGOverall order: PS < S < NPSG Significant difference between NPSG and PSSignificant difference between NPSG and PS Marginal difference between PS and SMarginal difference between PS and S

By-product: Conversational data collected from human By-product: Conversational data collected from human tutoringtutoring

Difference between tutorsDifference between tutors One of the tutors was very impatient and terse.One of the tutors was very impatient and terse.

Most of his students learnt less than expected from pre-test!Most of his students learnt less than expected from pre-test!

>> Content/presentation of tutorial dialog is critical to success>> Content/presentation of tutorial dialog is critical to success

88/44/44

CycleTalk: History

99/44/44

CycleTalk: HistoryCycleTalk: History

Since we had some Since we had some conversational data conversational data now:now: Analysis of TopicsAnalysis of Topics

Using Info-MagnetsUsing Info-Magnets

Arguello et. al. 2006Arguello et. al. 2006

15 topics identified15 topics identified Each human-tutoring Each human-tutoring

log, covered between 4 log, covered between 4 to 11 of these topicsto 11 of these topics

Correlation between Correlation between coverage of these coverage of these topics on learning from topics on learning from the human-tutoring the human-tutoring datadata

So, lets get the machine So, lets get the machine to do the talking.to do the talking.

1010/44/44

CycleTalk: HistoryCycleTalk: History Developing a Tutorial Developing a Tutorial

Dialog Systems for Dialog Systems for CycleTalkCycleTalk

12 Knowledge 12 Knowledge Construction Dialogs Construction Dialogs authored based on this authored based on this topics identifiedtopics identified

7 related to optimization 7 related to optimization relationshipsrelationships

5 related to theoretical 5 related to theoretical conceptsconcepts

Authored for TuTalkAuthored for TuTalk

When do we bring these When do we bring these up?up?

Tutor: Let’s stop for a minute and reflect on what we have seen. Having seen how changing T-Max (i.e., the Maximum Temperature at the Heater Outlet) affects the efficiency of the cycle, do you think it is a good idea in general for the Maximum Temperature to be increased and kept high?Student: yes

Tutor: Okay. Let’s think now about one important consequence of keeping T-Max high. What happens to the steam quality at the turbine exit if you in-crease the maximum temperature of the cycle?Student: Increases

Tutor: Right. As increasing the temperature makes the efficiency and steam quality better, we should increase the maximum temperature of the cycle. But can we keep on increasing the maximum temperature at the heater outlet indefinitely, or are there limitations to keep in mind?Student: there are limitations such as material constraints

Tutor: Great. we should try to keep T-Max (i.e., the maximum temperature at the Heater outlet of the cycle) as high as possible without risking safety concerns or exceeding the maximum temperature the material can withstand. Keeping T-Max high increases efficiency and keeps the steam quality high, which are both important. This is an important principle to keep in mind while optimizing the efficiency of your cycle.

1111/44/44

CycleTalk: HistoryCycleTalk: History Developing a Tutorial Developing a Tutorial

Dialog Systems for Dialog Systems for CycleTalkCycleTalk

12 Knowledge 12 Knowledge Construction Dialogs Construction Dialogs authored based on this authored based on this topics identifiedtopics identified

7 related to optimization 7 related to optimization relationshipsrelationships

5 related to theoretical 5 related to theoretical conceptsconcepts

Authored for TuTalkAuthored for TuTalk

Integrated with the Integrated with the example tracing tutor at example tracing tutor at relevant nodesrelevant nodes So, instead of hints, a So, instead of hints, a

window with the dialog window with the dialog agent would pops upagent would pops up Not a clean integrationNot a clean integration

1212/44/44

CycleTalk: HistoryCycleTalk: History Ran an experiment with our first CycleTalk with Tutorial Ran an experiment with our first CycleTalk with Tutorial

DialogDialog

Kumar et. al., Evaluating the Effectiveness of Kumar et. al., Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Tutorial Dialogue Instruction in an Exploratory Learning Context, 2006Learning Context, 2006 3 conditions3 conditions

S – Script onlyS – Script only PSHELP – PS + Tutorial Dialog triggered in place of some PSHELP – PS + Tutorial Dialog triggered in place of some

hintshints PSSUCCESS – PSHELP + Tutorial Dialog triggered on PSSUCCESS – PSHELP + Tutorial Dialog triggered on

successful completion of certain trace nodessuccessful completion of certain trace nodes

Effect sizeEffect size CMU: 0.35CMU: 0.35σσ comparing PSHELP to PSSUCCESS comparing PSHELP to PSSUCCESS USNA: 0.25USNA: 0.25σσ comparing S to PSSUCCESS comparing S to PSSUCCESS

Average KCD launches:Average KCD launches: PSHELP - 1.8 , PSSUCCESS – 2.7PSHELP - 1.8 , PSSUCCESS – 2.7 Human Tutoring – 4 to 11Human Tutoring – 4 to 11

1313/44/44

CycleTalk: Fall 2006CycleTalk: Fall 2006 Collaborative Learning Setup: ImplementationsCollaborative Learning Setup: Implementations

Interaction Model: Keep the students more engaged:Interaction Model: Keep the students more engaged: Hinting promptsHinting prompts

“Try to think of an idea related to manipulating a property of the pump.” Every one minute (too many!)Every one minute (too many!) Targeted prompting (based on contribution rate)Targeted prompting (based on contribution rate)

Dynamic/Adaptive support for collaboration vs. Dynamic/Adaptive support for collaboration vs. ScriptingScripting Topic Filter for triggering of dialogsTopic Filter for triggering of dialogs

Trained models to classify turns into topics (Taghelper)Trained models to classify turns into topics (Taghelper) Training data from Human-Tutoring corpusTraining data from Human-Tutoring corpus

(Rosé et. al., 2005)(Rosé et. al., 2005) 2 step classification2 step classification

1.1. Topic worthiness: SVM Topic worthiness: SVM (Q: How do we know if worthy?)(Q: How do we know if worthy?)2.2. Topic labeling: TDIDF ScoringTopic labeling: TDIDF Scoring

Chatting softwareChatting software Agent as an observer/participant in chatAgent as an observer/participant in chat

1414/44/44

CycleTalk: ImplementationsCycleTalk: Implementations

1515/44/44

CycleTalk: ProcedureCycleTalk: Procedure 15min: 15min: CyclePad trainingCyclePad training (led by experimenter)(led by experimenter)

70min: 70min: Work through material on domain Work through material on domain content and using CyclePadcontent and using CyclePad

15min: 15min: Pre-TestPre-Test 42 multiple choice, 8 open response questions42 multiple choice, 8 open response questions

Contest announcementContest announcement 25min: 25min: Review and write notes about what Review and write notes about what

students learnt into the chat windowstudents learnt into the chat window 10min: 10min: Planning/Synthesis of 2 design plansPlanning/Synthesis of 2 design plans 25min: 25min: Implementation of designs in Implementation of designs in

CyclePadCyclePad 15min: 15min: Post-TestPost-Test QuestionnaireQuestionnaire (Pairs only) (Pairs only)

1616/44/44

CycleTalk: Experimental DesignCycleTalk: Experimental Design Manipulation in step 4: Manipulation in step 4: Review and write Review and write

notes about what students learnt into notes about what students learnt into the chat windowthe chat window

3x2 Full factorial design3x2 Full factorial design SupportSupport

None (N): No supportNone (N): No support Static (S): Written material (Script)Static (S): Written material (Script) Dynamic (D): Adaptive dialog agentDynamic (D): Adaptive dialog agent

Collaboration: Alone (I) / Pair (P)Collaboration: Alone (I) / Pair (P)

CMU Sophomores: 87 students over 4 daysCMU Sophomores: 87 students over 4 days

1717/44/44

CycleTalk: Outcome MetricsCycleTalk: Outcome Metrics Pre/Post TestsPre/Post Tests

Objective type questionsObjective type questions Open response questionsOpen response questions

Ability to design, implement an efficient Ability to design, implement an efficient Ranking CycleRanking Cycle

QuestionnaireQuestionnaire

1818/44/44

CycleTalk: Results: CollaborationCycleTalk: Results: Collaboration

Effect size: 0.4Effect size: 0.4σσ (Q: Does this mean significant?)(Q: Does this mean significant?) Reflection in pairs is more effective than Reflection in pairs is more effective than

reflection alonereflection alone

1919/44/44

CycleTalk: Results: SupportCycleTalk: Results: Support

Positive effect of Dynamic support Positive effect of Dynamic support Dynamic Support > No supportDynamic Support > No support Effect size: 0.7Effect size: 0.7σσ

2020/44/44

CycleTalk: Results: CombinedCycleTalk: Results: Combined

Marginal Interaction, Marginal Interaction, p=.07p=.07

Pair+Dynamic > Pair+Dynamic > Individual+No Individual+No Support, 1.2Support, 1.2σσ

Pair+Static > Pair+Static > Individual+No Individual+No Support, .9Support, .9σσ

Individual+Dynamic Individual+Dynamic > Individual+No > Individual+No Support, 1.06Support, 1.06σσ

2121/44/44

CycleTalk: More ResultsCycleTalk: More Results

Open response questionsOpen response questions Advantage for dialog based support, but not collaborationAdvantage for dialog based support, but not collaboration Effect size: 0.5Effect size: 0.5σσ (Simpler ANCOVA model) (Simpler ANCOVA model)

Practical Assessment:Practical Assessment: No significant differencesNo significant differences 91% students built one fully defined cycle91% students built one fully defined cycle 64% built two64% built two

Questionnaire:Questionnaire: Dynamic support student rate high on benefit, but low on Dynamic support student rate high on benefit, but low on

engagement when in pairsengagement when in pairs Collaboration & Dynamic support not working together??Collaboration & Dynamic support not working together?? Desirable Difficulty? (Ref: Robert Bjork)Desirable Difficulty? (Ref: Robert Bjork)

2222/44/44

CycleTalk: ObservationsCycleTalk: Observations

Individual interaction with agentsIndividual interaction with agents Highly tutor directedHighly tutor directed Students rarely ignore tutor promptsStudents rarely ignore tutor prompts

More complicated dynamics with pairsMore complicated dynamics with pairs Students talk around tutor agent Students talk around tutor agent (Q)(Q) Tutor agent is an interruptionTutor agent is an interruption Students treat agent contributions like hintsStudents treat agent contributions like hints Evidence that they read although they often don’t Evidence that they read although they often don’t

respondrespond FrustrationFrustration

2323/44/44

AIED 2007: Your questions (Q) Is this reflection?

Dialogs were about material already given to them which they review in step 2

Whether students were receiving instruction twice?

Multi-Party Turn-Taking This was a problem back then

Recent attempts to improve this And more needs to be done…

Why do students ignore the Tutors?

Pairwise Tukey Posthoc analysis Difference of means/SE > qcritical

2424/44/44

That was Fall 2006: Since then…That was Fall 2006: Since then… Making the agent more “Social”Making the agent more “Social”

Kumar et. al., SLaTE 2007Kumar et. al., SLaTE 2007 Engaging students through small talkEngaging students through small talk Promoting CollaborationPromoting Collaboration

Unpublished Manuscript (Kumar et. al.)Unpublished Manuscript (Kumar et. al.) Attention GrabbingAttention Grabbing

Chaudhuri et. al., AIED 2009Chaudhuri et. al., AIED 2009 PointersPointers

BasilicaBasilica Architecture for developing conversational agentsArchitecture for developing conversational agents

Virtual EnvironmentsVirtual Environments ConcertChatConcertChat SecondLifeSecondLife

More than 2 students?More than 2 students? Recent work in a freshmen Mechanical Engineering classRecent work in a freshmen Mechanical Engineering class

2525/44/44

Kumar et. al., SLaTE 2007Kumar et. al., SLaTE 2007 Mathematics Problem SolvingMathematics Problem Solving

FractionsFractions Addition, Subtraction, Multiplication, DivisionAddition, Subtraction, Multiplication, Division

Conversation Interface(shared by students & (shared by students & conversational conversational agents)agents)

CTAT Problem Solving Interface with Cognitive Tutors based feedback [Aleven, Koedinger](shared by students)(shared by students)

2626/44/44

Kumar et. al., SLaTE 2007: Social Kumar et. al., SLaTE 2007: Social StrategyStrategy

Tutor: Student1, If you had to choose between a long flight and a longcar ride, which seems less uncomfortable?

Student1: I’ll take the car ride

Tutor: Ok Student1.

Tutor: Student2, Which are more entertaining – books or movies?

Student2: definitely books!

Tutor: Ok Student2.

Tutor: Please work on the problem shown in the side panel.

Motivated by the intention of Motivated by the intention of engaging studentsengaging students By showing interest in By showing interest in

their personal preferencestheir personal preferences

Goal of this social Goal of this social conversation is to make conversation is to make the students feel that they the students feel that they worked together to worked together to construct the problem construct the problem statementstatement

Comes up every time the Comes up every time the students are about to start students are about to start solving a new problemsolving a new problem

Related: (Bickmore & Related: (Bickmore & Cassell)Cassell) SmallTalk by Embodied SmallTalk by Embodied

Conversational Agent REAConversational Agent REA

Example

Jan packed several books to amuse Jan packed several books to amuse herself on a long car ride to visit her herself on a long car ride to visit her grandma. After 1/5 of the trip, she had grandma. After 1/5 of the trip, she had already finished 6/8 of the books she already finished 6/8 of the books she brought. How many times more books brought. How many times more books should she have brought than what she should she have brought than what she packed?packed?

2727/44/44

Kumar et. al., SLaTE 2007: QuestionnaireKumar et. al., SLaTE 2007: Questionnaire

Control Control ExperimentExperimentalal

Perceived Self CompetencePerceived Self Competence 4.2 (.56)4.2 (.56) 4.1 (.23) ns4.1 (.23) ns

Perceived Partner CompetencePerceived Partner Competence 4.3 (.62)4.3 (.62) 3.9 (.49) ns3.9 (.49) ns

Perceived Benefit of Perceived Benefit of CollaborationCollaboration

4.5 (.74)4.5 (.74) 4.4 (.70) ns4.4 (.70) ns

Perceived Help Received*Perceived Help Received* 1.81.8 (1.3)(1.3)

3.33.3 (.69) (.69)

Perceived Help Provided*Perceived Help Provided* 1.81.8 (1.1)(1.1)

3.13.1 (1.1) (1.1)

Significant (Effect Size = 1.15)Significant (Effect Size = 1.15)Students perceived higher help offering Students perceived higher help offering

by their partners in the Experimental by their partners in the Experimental conditioncondition

Significant (Effect Size = 1.18)Significant (Effect Size = 1.18)Students perceived they offered more Students perceived they offered more

help tohelp totheir partners in the Experimental their partners in the Experimental

conditioncondition

2828/44/44

Observations from Conversation AnalysisObservations from Conversation Analysis

Average number of Help Provisions not significantly different Average number of Help Provisions not significantly different across conditionsacross conditions

More help related episodes per problem in the Experimental More help related episodes per problem in the Experimental conditioncondition

Mean (Control) = 0.30Mean (Control) = 0.30 Mean Mean (Experimental) = 0.69(Experimental) = 0.69

F(1, 15) = 16.8F(1, 15) = 16.8 p < 0.001 p < 0.001

More episodes of Deny Help in Control conditionMore episodes of Deny Help in Control conditionMean (Control) = 40.2Mean (Control) = 40.2 Mean (Experimental) = 24.7 Mean (Experimental) = 24.7F(1, 62) = 3.46F(1, 62) = 3.46 p = 0.001 p = 0.001

Students displayed more negative attitude in Control Students displayed more negative attitude in Control conditionsconditions Insults (“you stink”, “stupid”) occurred only in Control Insults (“you stink”, “stupid”) occurred only in Control

conditioncondition

Kumar et. al., SLaTE 2007: ResultsKumar et. al., SLaTE 2007: Results

2929/44/44

That was Fall 2006: Since then…That was Fall 2006: Since then… Making the agent more “Social”Making the agent more “Social”

Kumar et. al., SLaTE 2007Kumar et. al., SLaTE 2007 Engaging students through small talkEngaging students through small talk Promoting CollaborationPromoting Collaboration

Unpublished Manuscript (Kumar et. al.)Unpublished Manuscript (Kumar et. al.) Attention GrabbingAttention Grabbing

Chaudhuri et. al., AIED 2009Chaudhuri et. al., AIED 2009 PointersPointers

BasilicaBasilica Architecture for developing conversational agentsArchitecture for developing conversational agents

Virtual EnvironmentsVirtual Environments ConcertChatConcertChat SecondLifeSecondLife

More than 2 students?More than 2 students? Recent work in a freshmen Mechanical Engineering classRecent work in a freshmen Mechanical Engineering class

3030/44/44

CycleTalk Fall 2007CycleTalk Fall 2007

3131/44/44

CycleTalk Fall 2007CycleTalk Fall 2007

Instructive Conversationwith Attention Grabbing

Now might be a good time forreflection.Tutor

well the idea is to increase the heatinSt13BP

Power is generated when steam iscooled. If there is more heat input,will there be more or less potentialfor cooling?

Tutor

because that increases powerSt13BPThere will be more potential forCooling. If there is more potentialfor cooling, is there more or lesspotential for power generation?

Tutor

Consider the effect of increasing Qinwhich is heat input to a cycle. First,let’s consider what happens topower out when Qin is increased.What happens to power out whensteam is cooled?

Tutor

Two Motivational Prompts

Solving this problem can be a lot offun if you cooperate well togetherTutor

At 2 minute mark

Winning isn’t everything.Don’t worry. Be Happy.Tutor

At 30 minute mark

Instructive Conversationwith Attention Grabbing

Now might be a good time forreflection.Tutor

well the idea is to increase the heatinSt13BP

Power is generated when steam iscooled. If there is more heat input,will there be more or less potentialfor cooling?

Tutor

because that increases powerSt13BPThere will be more potential forCooling. If there is more potentialfor cooling, is there more or lesspotential for power generation?

Tutor

Consider the effect of increasing Qinwhich is heat input to a cycle. First,let’s consider what happens topower out when Qin is increased.What happens to power out whensteam is cooled?

Tutor

3232/44/44

CycleTalk Fall 2007: CycleTalk Fall 2007: Conversation Conversation AnalysisAnalysis

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s

Tutoring without Tutoring without Attention Attention Grabbing Grabbing

Tutoring with Tutoring with Attention Attention Grabbing Grabbing

ResponseResponseto Instructional to Instructional

TurnsTurns2.1%2.1% 12.5%12.5%

Relevant ResponseRelevant Responseto Instructional to Instructional

TurnsTurns1.1%1.1% 8.6%8.6%More More

Attention to Attention to the Tutorthe Tutor

More More Relevant Relevant

ResponsesResponses

Numbers here are:Numbers here are:Average ( Percentage of Average ( Percentage of

Instructional turns each student Instructional turns each student responded to)responded to)

3333/44/44

CycleTalk Fall 2007: CycleTalk Fall 2007: Conversation Conversation AnalysisAnalysis

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s

WithoutWithoutMotivationalMotivational

PromptsPrompts

WithWithMotivationalMotivational

PromptsPrompts

ReferenceReferenceto the Tutorto the Tutor

0.300.30± 0.65± 0.65

0.840.84± 1.25± 1.25

Positive ReferencePositive Referenceto the Tutorto the Tutor 10%10% 45%45%

3434/44/44

CycleTalk Fall 2008 CycleTalk Fall 2008 (Chaudhuri et. al. (Chaudhuri et. al. ‘09)‘09)B6: so i think the best thing to go with first is to increase the pressure entering the connector

Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book.

Tutor: type HELP WITH CHANGEPMAX if you want to discuss it with me.

B1: help with changepmax

Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer to the figure that I have pasted onto your whiteboard for your convenience. Having seen how changing P-Max, the Maximum Pressure at the Heater Inlet, effects efficiency and Steam, do you think as a general principle the Maximum pressure should be increased ?

B6: Yes

Tutor: So the Efficiency of a Cycle increases by increasing the maximum pressure of the cycle. In addition to the effect on efficiency, what happens to the steam quality at the turbine exit if you increase the maximum pressure of the cycle ?

B6: Decreases

Tutor: So let's think for a minute about these competing effects: Increasing P-Max makes the efficiency better but the steam quality degrades. Is the steam quality at the turbine exit acceptable at the maximum allowable pressure within safety limits?

3535/44/44

CycleTalk Fall 2008: ExperimentCycleTalk Fall 2008: Experiment Manipulation (2x2):Manipulation (2x2):

Pointer Hints: Yes/NoPointer Hints: Yes/No Dialog Support: Yes/NoDialog Support: Yes/No

ResultsResults Higher learning gains for Pointer+Dialog Higher learning gains for Pointer+Dialog

conditioncondition Pointer+Dialog vs Dialog: 0.8Pointer+Dialog vs Dialog: 0.8σσ Pointer+Dialog vs None: 0.6Pointer+Dialog vs None: 0.6σσ Pointer vs None: 0.35Pointer vs None: 0.35σσ

Few dialogs in Pointer + Dialog condition compared to Few dialogs in Pointer + Dialog condition compared to Dialog only conditionDialog only condition Too many dialogs distracting?Too many dialogs distracting?

3636/44/44

That was Fall 2006: Since then…That was Fall 2006: Since then… Making the agent more “Social”Making the agent more “Social”

Kumar et. al., SLaTE 2007Kumar et. al., SLaTE 2007 Engaging students through small talkEngaging students through small talk Promoting CollaborationPromoting Collaboration

Unpublished Manuscript (Kumar et. al.)Unpublished Manuscript (Kumar et. al.) Attention GrabbingAttention Grabbing

Chaudhuri et. al., AIED 2009Chaudhuri et. al., AIED 2009 PointersPointers

BasilicaBasilica Architecture for developing conversational agentsArchitecture for developing conversational agents

Virtual EnvironmentsVirtual Environments ConcertChatConcertChat SecondLifeSecondLife

More than 2 students?More than 2 students? Recent work in a freshmen Mechanical Engineering classRecent work in a freshmen Mechanical Engineering class

3737/44/44

BasilicaBasilica Multi-Expert model of building Multi-Expert model of building

conversational agentsconversational agents

Terminology:Terminology: Components, Actors, Filters, Events, Components, Actors, Filters, Events,

ConnectionsConnections

Actors: Generate user perceivable eventsActors: Generate user perceivable events Filters: (Everything else, mostly): Interpret Filters: (Everything else, mostly): Interpret

events generated by other componentsevents generated by other components

Data is encapsulated as EventsData is encapsulated as Events

3838/44/44

BasilicaBasilica

TuTalk Server

ConcertChatServer

Student1

Student2

PresenceActor

TextMessageEvent

x

x

TextMessageEvent

ChannelFilter

TextMessageEvent

TextMessageEvent

OutGoingMessageSpoolingFilter

TextMessageEvent

TextMessageEvent

CCTextFilter

x

LaunchEvent

LaunchFilter

TookTutorTurnEventTextMessageEvent

TakeTutorTurnEvent

TurnTakingFilter

GrabAttentionEventTextMessageEvent

GrabAttentionEventAttentionGrabbed-

Event

AttentionGrabbing

Filter TakeTutorTurnEventTookTutorTurnEventLaunchEventTutoringStartedEventDoneTutoringEventAttentionGrabbedEvent

TakeTutorTurnEventTookTutorTurnEvent

StartTutoringEventGrabAttentionEvent

TutoringFilter

LaunchEventTextMessageEvent

ProduceHintEvent

HintingFilter

Polling

PromptingActor

LaunchEvent

TextMessageEvent

HintingActor

ProduceHintEvent

TextMessageEvent

TutoringActor

StartTutoringEventTakeTutorTurnEvent

TextMessageEventTutoringStartedEventTookTutorTurnEvent

DoneTutoringEvent

AttentionGrabbing

Actor

GrabAttentionEvent

TextMessageEvent

3939/44/44

BasilicaBasilica Novel ArchitectureNovel Architecture

Re-usable componentsRe-usable components Rapid prototypesRapid prototypes

Easy integration of the same agent with many Easy integration of the same agent with many environmentsenvironments

Incremental developments of componentsIncremental developments of components

Meta-architecture for bringing together other Meta-architecture for bringing together other dialog management componentsdialog management components

4040/44/44

Conversational Agents in Second Conversational Agents in Second LifeLife

Session1 Session2 Session3 Session4 Session5 …

MIDDLEWARE

TRANSLATION

Object 1 Internal Representation

Message Receiver

MessageQueue

Object 2 Internal Representation

Message Receiver

MessageQueue

INTERFACE

OBJECT1

INTERFACE

OBJECT2

HT

TP H

TT

P

4141/44/44

That was Fall 2006: Since then…That was Fall 2006: Since then… Making the agent more “Social”Making the agent more “Social”

Kumar et. al., SLaTE 2007Kumar et. al., SLaTE 2007 Engaging students through small talkEngaging students through small talk Promoting CollaborationPromoting Collaboration

Unpublished Manuscript (Kumar et. al.)Unpublished Manuscript (Kumar et. al.) Attention GrabbingAttention Grabbing

Chaudhuri et. al., AIED 2009Chaudhuri et. al., AIED 2009 PointersPointers

BasilicaBasilica Architecture for developing conversational agentsArchitecture for developing conversational agents

Virtual EnvironmentsVirtual Environments ConcertChatConcertChat SecondLifeSecondLife

More than 2 students?More than 2 students? Recent work in a freshmen Mechanical Engineering classRecent work in a freshmen Mechanical Engineering class

4242/44/44

Supporting GroupsSupporting Groups Mechanical Engineering Freshmen classMechanical Engineering Freshmen class

Sort of repeat the process of CycleTalkSort of repeat the process of CycleTalk Initial data collection done with agents thoughInitial data collection done with agents though One advantage of Basilica hereOne advantage of Basilica here

Observations: (3 out of 6 sessions)Observations: (3 out of 6 sessions) New vocabulary collection (for dynamic triggers)New vocabulary collection (for dynamic triggers) Lots of Sarcasm, Teasing, Cursing, Discontent, SillinessLots of Sarcasm, Teasing, Cursing, Discontent, Silliness

Some positiveness tooSome positiveness too Abuses towards TutorAbuses towards Tutor Opportunities identified that next version of tutor can useOpportunities identified that next version of tutor can use

Current irresponsivenessCurrent irresponsiveness frequent questions/conceptsfrequent questions/concepts

GRASPGRASP Supporting teachers/facilitators for long-term group Supporting teachers/facilitators for long-term group

assessmentassessment

4343/44/44

Our work: QuestionsOur work: Questions Supporting Learning tasksSupporting Learning tasks

What kind of tasks?What kind of tasks? What kind of support? (Agents ofcourse, but…)What kind of support? (Agents ofcourse, but…)

What role?What role? ManifestationManifestation How do we make sure the support is getting through?How do we make sure the support is getting through?

What kind of users?What kind of users? Individuals / Pairs / Groups ?Individuals / Pairs / Groups ? Students ? Teachers/facilitators?Students ? Teachers/facilitators?

What environment?What environment? How do we build this support?How do we build this support? How do we evaluate if the support helps?How do we evaluate if the support helps?

4444/44/44

Done:Done:

Thanks for tuning in.Thanks for tuning in.

Most questions are Most questions are good questions.good questions.

So, Please Ask.So, Please Ask.

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s