Conversational AnalysisConversational Analysis

Concert e-Learning SystemConcert e-Learning SystemHICSS Workshop on Persistent ConversationHICSS Workshop on Persistent Conversation

Kona, HawaiiKona, HawaiiJanuary 3, 2007January 3, 2007John C. ThomasJohn C. Thomas

IBM T. J. Watson Research CenterIBM T. J. Watson Research Center

OutlineOutline

• Dynamic Learning EnvironmentDynamic Learning Environment

• General Quality of Learning in ConcertGeneral Quality of Learning in Concert

• Quantitative Analysis of CommentsQuantitative Analysis of Comments

• Level of LearningLevel of Learning

• Critical Incidents and Missed Critical Incidents and Missed OpportunitiesOpportunities

• Suggestions for Systems to Support Suggestions for Systems to Support LearningLearning

Initial Qualitative Initial Qualitative ObservationsObservations

• Uncertainty about and awkwardness in using tool Uncertainty about and awkwardness in using tool pervades conversation pervades conversation – Leads to group-bonding humorLeads to group-bonding humor– Leads to division of laborLeads to division of labor– ““Facilitator” interrupts flow of problem solving with Facilitator” interrupts flow of problem solving with

technology issuestechnology issues

• Uncertainty about what the problem isUncertainty about what the problem is• Uncertainty about times and participantsUncertainty about times and participants• Conversation seems to be “tip of the iceberg”Conversation seems to be “tip of the iceberg”• Real names Real names politeness & inclusion ? Hard to know politeness & inclusion ? Hard to know

complete context but this is very “nice talk” for complete context but this is very “nice talk” for InternetInternet

• Medium seems conducive to small “snippets” of Medium seems conducive to small “snippets” of thought rather than extended problem solvingthought rather than extended problem solving

• Actual Problem Solving (Wang) apparently takes Actual Problem Solving (Wang) apparently takes place individually and off-lineplace individually and off-line

Dynamic Learning Dynamic Learning EnvironmentEnvironment• Designed for adult professionalsDesigned for adult professionals• User entered query and learning User entered query and learning

constraints such as time and type of constraints such as time and type of material and chosen resultsmaterial and chosen results

• ““Custom Course” constructed from Custom Course” constructed from Learning Objects and ontologyLearning Objects and ontology

• Mechanisms for groups but focus on Mechanisms for groups but focus on individual learnerindividual learner

Dynamic Learning Dynamic Learning EnvironmentEnvironment• Experiment compared Query Only with Experiment compared Query Only with

Dynamic Learning Environment (DLE)Dynamic Learning Environment (DLE)• DLE users spent more time on fewer DLE users spent more time on fewer

objectsobjects• Users in Dynamic Learning Environment Users in Dynamic Learning Environment

made more extensive and better designsmade more extensive and better designs• Verbal behavior had more “cognitive” Verbal behavior had more “cognitive”

words; fewer “search and find” wordswords; fewer “search and find” words• Use of DLE encouraged learning “set” and Use of DLE encouraged learning “set” and

gave overall structure gave overall structure

0

50

100

150

200

250

300

350

P T C S O M D

Freq.

P: Problem Solving

T: Technology Discussion

C: Clarification Dialogue

S: Sociality: apology, thanks, greetings, goodbye, praise, emotion,etc.

O: Okay, nods, short agreement with previous speaker

D: Direction: problem selection and formulation

M: Meeting logistics: who is here; when do we meet, etc.

First “Half” First “Half” Social Social CommentsComments

Second “Half” Second “Half” Social Social CommentsComments

8.02 %8.02 % 23.0 %23.0 %

MM DD TT CC PP SS

MM 131131 2828 66 77 99 4040

DD 2323 191191 1818 2323 2727 2828

TT 77 1717 130130 33 77 1010

CC 1010 1818 77 102102 2222 99

PP 99 2828 77 2222 9797 1717

SS 4040 3131 99 99 1414 131131

P: Problem SolvingT: Technology DiscussionC: Clarification Dialogue S: Sociality: apology, thanks, greetings, goodbye, praise, emotion,etc. D: Direction: problem selection and formulationM: Meeting logistics: who is here; when do we meet, etc.

CODECODE Longest StringLongest String

D (direction)D (direction) 2424

T (technology)T (technology) 2121

C (clarification)C (clarification) 1515

S (social)S (social) 1515

M (meeting logistics)M (meeting logistics) 1414

P (problem solving)P (problem solving) 1010

Longest “string” of one type of comment with no more than one continuous string of interruptions

745 Wang plz rmb to type this formula carefully 07.06.2006 09.34.04   D

746 Wang it is 9+(N-2)*3 07.06.2006 09.34.25   P

747 nan you did good job at putting your findings on the whiteboard, which makes it easier to summarize your work

07.06.2006 09.34.29 Reference to message No. 742 (Text: "so clarice, can you wrap this up and finish it?") R

271 Amanda2 oh.....you know ...the others have come up with complicated formulas.i think we're doing the thing directly.i'm not sure if we're supposed to solve the question just like that?

26.05.2006 09.07.19   D

928 Wang so how is it with the drawing 08.06.2006 08.44.23   C

929 Wang are the octogons successfully drawn 08.06.2006 08.44.38   C

930 Clarice2 wait 08.06.2006 08.44.49   C

931 Clarice2 something is wrong 08.06.2006 08.44.50   P

932 dchia yep cause they cant join on all sides properly 08.06.2006 08.45.05   P

933 Wang sorry 08.06.2006 08.45.11   R

934 Amanda2 yea 08.06.2006 08.45.14   O

935 Clarice2 yes 08.06.2006 08.45.20   O

936 Wang i think octogon is not suitabe for this prob 08.06.2006 08.45.23   D

937 Clarice2 so that means we cannot use octagons? 08.06.2006 08.45.30   C

938 Wang so why dun we stick with hexagon 08.06.2006 08.45.31   D

Possible continuations…Possible continuations…

• What is it about octagons that makes What is it about octagons that makes them not suitable?them not suitable?

• Which shapes “work” for this type of Which shapes “work” for this type of problem?problem?

• Does the fact that they don’t pack Does the fact that they don’t pack evenly (really) mean we cannot find evenly (really) mean we cannot find a method or formula?a method or formula?

952 Wang i have racked my brain yesterdAY TO only come up with that formula 08.06.2006 08.48.25   D

Possible continuations….Possible continuations….

• Algorithmic generation versus Algorithmic generation versus mathematical formula: what are the mathematical formula: what are the advantages and disadvantages?advantages and disadvantages?

• Can all formulae be described as Can all formulae be described as algorithms? Vice versa?algorithms? Vice versa?

1044 Clarice2 why don't we try substitutinf life situations in the patterns instead of shapes

08.06.2006 09.10.44   D

1045 dchia eh guys? sorry bt i really dont know what u mean by the multiples of 3 method..

08.06.2006 09.11.03   C

1046 Clarice2 see ar 08.06.2006 09.11.23   C

1047 Clarice2 look under pattern 1 and 2 08.06.2006 09.11.30   C

1048 Clarice2 the difference between the number of sticks is 9 right? 08.06.2006 09.11.43   C

1049 dchia ya 08.06.2006 09.11.50   C

Possible continuations…Possible continuations…

• Are you joking? What did you mean?Are you joking? What did you mean?

• How could we model life situations in How could we model life situations in terms of geometry?terms of geometry?

• Life situations? Like what?Life situations? Like what?

• Life situations? You mean like human Life situations? You mean like human relationships? relationships?

1385 Clarice2 ok then 09.06.2006 08.37.16   C

1386 Amanda2 oh oh clarice 09.06.2006 08.37.29   M

1387 Amanda2 when i count i leave out the sticks next to it 09.06.2006 08.37.37   C

1388 Amanda2 i don't think we shd include htem 09.06.2006 08.37.44   D

1389 Amanda2 coz who puts two sticks at the same place 09.06.2006 08.37.53   D

1390 Clarice2 ya i noie 09.06.2006 08.38.07   O

Possible continuations…Possible continuations…

• Do “extra” sticks make structures Do “extra” sticks make structures stronger? Under what conditions?stronger? Under what conditions?

• What is the relationship among our What is the relationship among our diagrams, mathematical diagrams, mathematical abstractions, and real world objects?abstractions, and real world objects?

• Would the “extra” sticks be useful if Would the “extra” sticks be useful if we were building objects in a virtual we were building objects in a virtual world? Under what conditions?world? Under what conditions?

Design Thoughts…Design Thoughts…

• Use Socio-Technical Pattern LanguageUse Socio-Technical Pattern Language

• Special RolesSpecial Roles– Learning FacilitatorLearning Facilitator– ScribeScribe

• Process SupportProcess Support– Stages: Problem Finding, Problem Formulation, Stages: Problem Finding, Problem Formulation,

Problem Solving, Reflection: What did we Problem Solving, Reflection: What did we learn? How could we solve such problems more learn? How could we solve such problems more effectively in the future?effectively in the future?

ReferencesReferences

• Farrell, R., Liburd, S.D. and Thomas, J.C. Dynamic Assembly of Farrell, R., Liburd, S.D. and Thomas, J.C. Dynamic Assembly of Learning Objects. Learning Objects. Proceedings of the World-Wide Web ConferenceProceedings of the World-Wide Web Conference, , (2004) New York, NY.(2004) New York, NY.

• Thomas, J. C. and Farrell, R. G. An experimental investigation on the Thomas, J. C. and Farrell, R. G. An experimental investigation on the effectiveness of individualized web-based learning based on the effectiveness of individualized web-based learning based on the dynamic assembly of learning objects, dynamic assembly of learning objects, IBM Technical Report, IBM Technical Report, (2004)RC 23338.(2004)RC 23338.

• Dynamic Learning Experience. Dynamic Learning Experience. http://http://www.alphaworks.ibm.com/tech/dlewww.alphaworks.ibm.com/tech/dle

• Farrell, R., Liburd, S., and Thomas, J. “Dynamic Assembly of Learning Farrell, R., Liburd, S., and Thomas, J. “Dynamic Assembly of Learning Objects”, Objects”, Proceedings of the 13th International World-Wide Web Proceedings of the 13th International World-Wide Web ConferenceConference (New York, NY, May 2004). (New York, NY, May 2004).

• Brookfield, S., D. (1986) Understanding Brookfield, S., D. (1986) Understanding and Facilitating Adult and Facilitating Adult LearningLearning. San Francisco: Jossey-Bass. . San Francisco: Jossey-Bass.

• Carroll, J. M. and Mack, R. L. (1984). Learning to use a word Carroll, J. M. and Mack, R. L. (1984). Learning to use a word processor: By doing, by thinking, by knowing. In J. C. Thomas & M. L. processor: By doing, by thinking, by knowing. In J. C. Thomas & M. L. Schenider (Eds.), Schenider (Eds.), Human factors in computer systems. Human factors in computer systems. Norwood, N.J.: Norwood, N.J.: Ablex.Ablex.

• Ausubel, D.P. (1960). The use of advance organizers in the learning Ausubel, D.P. (1960). The use of advance organizers in the learning and retention of meaningful verbal material. and retention of meaningful verbal material. Journal of Educational Journal of Educational PsychologyPsychology, , 5151, 267-272., 267-272.