analysis of interaction in collaborative activities; the synergo approach

1/60

Analysis of interaction in collaborative activities:

the Synergo trail

Nikolaos AvourisUniversity of Patras, GR

Keynote Talk

INCoS 2010 – Thessaloniki November 24th

2/60

outline- on analysis of collaboration- the synergo testbed- synergo studies- models from synergo data

3/60

On analysis of collaborative

activities

4/60

Typical analysis objectivesfocusmethod

Participant’s perceptionsInquiry methods

Interaction processQuantitative, qualitative, sequential methods

Learning outcomesPre-post testing

Collaborative technologyUsability evaluation

5/60

Focus on the interaction process

– Dillenbourg: “the basic instrument for understanding collaborative learning is understanding the interaction that takes place during a learning process”

– Koschmann: “CSCL research is not focused on instructional efficacy, but it is studying instruction as enacted practice”

6/60

Quantitative analysis• Frequency counts of events such as:

- messages posted per student per period of time- hits on particular discussion forum pages - actions taken on objects of a shared workspace- number of files read in a shared file system etc.

• Defining metrics (indicators) that combine different kinds of frequency counts

• Suitable for all kinds of collaborative learning• They can lead to models of interaction (e.g.

Social Networks etc.)

7/60

Qualitative content analysis• “Content analysis refers to any process

that is a systematic replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding” (Kripendorf, 1980)

• Suitable for every means of dialogue oriented collaborative learning (synchronous & asynchronous, collocated & distant)

8/60

Content analysis models• Henri’s scheme• Garrison’s model• Gunawardena’s Interaction

Analysis Model• Language/action OCAF

9/60

Content analysis resources

• The content analysis guidebook http://academic.csuohio.edu/kneuendorf/content/

10/60

Small group synchronous interaction: Integration of dialogue and action• Treats language acts and actions taken to objects

in an integrated way• Uniform annotation (eg. the OCAF framework)• Shifts the focus to the objects of a shared

workspace• Objects have an ‘owner’ just like language acts• Can visualize uptaking actions (Suthers 05)

11/60

Dialogue: Chat tool affordances• Visual and/or auditory cues are not available• No production blocking->overlapping exchanges• Persistence of messages – substantiation of conversation• Loose inter-turn connectedness - but possibility of

simultaneous engagement in multiple threads• Verbal deixis spans throughout the whole history of

dialogue (no restricted time window is adequate for analysis)

• Posters may reply rapidly, using short messages and split long messages to increase referent/message coherency (Garcia and Jacobs 1999)

• Participants begin new topics fairly much at will in a manner that would not happen in a formal face-to-face group discussion (O’Neil & Martin, 2003)

12/60

Action: Shared Activity spaceaffordances• Feedthrough (Dix et. al., 1993)• Various degrees of coupling (Salvador

et. al., 1996)• Workspace can be used as an external

representation of the task that allows efficient nonverbal communication

• Workspace artefacts act as conversational props (Hutchins, 1990)

13/60

Types of communication acts / gestures in shared workspace

• Deictic references• Demonstrations • Manifesting actions• Visual evidence (Gutwin, Greenberg, 2002)

14/60

Grounding through actions on a workspace representation (Suthers, 2006)Sequences of actions :(1) one participant’s action in a

medium…(2) is taken up by another participant

in a manner that indicates understanding of its meaning, and

(3) the first participant signals acceptance

15/60

Merging Action and dialogue Annotated model=collection of objects (OCAF Avouris et al. 2003)

MEF = { Entities= E (ABC) = 1/EP, FA , EI

E (VELO) = 2/ EP, FA , EI E (TRUCK) = 3/FP, FI E (STOREHOUSE) = 4/FP EC, FA, FI E (STORE) = 5/FP EC, FA, FI Ε(DELIVERY)= 11/ FP, EX, FI

Relations= R (VELO-owns-SH) = 9/FPI R (VELO-owns-ST) = 10/FPI R(TRUCK-transports- DELIVERY)=17/ EP, FI, EC R(SH-are-suppplied-by-TR) = 18/ FIM R (ABC-owns-TR) = 25/ FPI R(ST-owns-SH) = 24/ EP FP FI EC, EM R (ABC-owns-TR) = 25/ FPI

Attributes= A (DEL.id) = 13/FIM A (DEL.volume) = 14/FIM A (DEL.Weight) = 15/FI A (DEL.Destination) = 16/FI A (TR.Max_Weight ) = 19/FI A (TR.id ) = 21/EP , FI A (TR.Journey_id ) = 23/FI A (TR.volume ) = 20FIM A (SH.id ) = 24/FI

Items not in the final solution -R (SH-DEL) = 12/EP , FR , -A(VELO.Storehouse)=6/ EP , FC -A(VELO.Store)= 7/ EP , FC -A(ABC.Truck)= 8/ FP , EX -A (TR.max_journeys_per_week) = 22/EP , FR }

A(volume)

A(destination)

A(Journey _id)

A(id) A(volume)

E(VELO)

2/EP, FA , EI

E(ABC) 1/EP, FA , EI

E(STORE-HOUSE)

4/FP EC, FI

E(STORE)

5/FP , EC, FAI

E(TRUCK) EP, FI

E(DE-LIVERY)

11/FP, EX, FI

20/FI,M

23/FI

21/EP, FI

14/FIM

16/FI

R

9/FPI

R 24/EP FPI, EM

A(id)

24/ FI

R

10/FPI

R 18/FIM

R 25/FPI

R 17/EP,FI,EC

R 12/EP, FR

A(Max_ weight)

19/FI

A(Id)

13/FIM

A(Weight

15/FI

A (max-journeys/week 22/EP, FR

A (storehouse)

6/EP, FC

A (store)

7/EP, FC A (truck) 8/FP, EX

16/60

Synergo

Avouris et al. 2004hci.ece.upatras.gr/synergo

Chat

Act

17/60

Synergo

Chat tool

Shared Activity Space

Drawing objects libraries

Partner selectiontool

18/60

Synergo Drawing librariesConcept maps

Flow charts

Entity-Relationship Diagrams

Free Drawing

19/60

Activity logging

used for :• Build a history of interaction at server• support latecomers during synchronous collaboration• analysis and playback of the activity •Support replication/ reduce bandwidth requirements

20/60

Analysis tools

20

21/60

Log Data Preprocessor

21

Analysistools

22/60

Typed events automatically annotate the diagram

Object A

I C

Actor A Actor B Actor C

Types of events I (Insert), M (Modify), D (Delete) C (Contest)

M R

( )itoaii TOAtE ][],[,,=

23/60

Playback of annotated view

24/60

What about the chat? Can we annotate chat automatically?

One approach is to ask the user to do it - open sentences (e.g. Epsilon (Soller et al. 97)

25/60

Abstract objects

Dialogue messages

Model objects

Deleted objects

(b)

Annotation of chat events

26/60

Define types of actions (annotation scheme)

27/60

Overview: Visualization of logged actions

28/60

Teachers view and tool support

• E. Voyiatzaki, M. Margaritis, N. Avouris, Collaborative Interaction Analysis: The teachers' perspective, Proc.ICALT 2006 - The 6th IEEE International Conference on Advanced LearningTechnologies. July 5-7, 2006 – Kerkrade , Netherlands, pp. 345-349.

29/60

Teacher support (supervisor tools)

30/60

Study of the use of tools by teachersComputer Engineering University degree program (A’ Semester)

Level of Education

1 Teacher + 5 Teaching Assistants Teachers involved

80 students (46 students 2004-2005, 34 students 2005-2006)

Learners involved

Problem solving activity: Development and Exploration of an Algorithm

Students in Dyads , no roles assignedTypical Laboratory lesson (2 didactic hours)

Collaborative Activity

SYNERGO Collaborative EnvironmentSYNERGO Analysis Tools

Collaborative Tools

31/60

teacher

The Teachers Used the proposed views and gave feedback…

Quantified Overview:Class and

Group

The ProcessView

(Playbackof the

activity)

Qualitativeview

Rowdata

researcher

Teachers: “The process view is the most important tool for in depth insight .”

32/60

studies

Vrachneika Gymnasio-3rd year

UnivPatras Algorithms

33/60

Typical tasks- Collaborative Cognitive Walkthrough of an interactive system

- Designing Data bases (ER-D)

- Building and exploring Flow Charts

34/60

Joint Univ Patras -UnivDuisburg croos-national collaborative activities (2004-2005)

• A. Harrer, G. Kahrimanis, S. Zeini, L. Bollen, N. Avouris, Is there a way to e-Bologna? Cross-National Collaborative Activities inUniversity Courses, Proceedings EC-TEL, Crete, October 1-4, 2006, LNCS vol. 4227/2006, pp. 140-154, Springer Berlin

35/60

Similar models with different tools (Synergo, Freestyler)

36/60

Findings of the Patras-Duisburg study

•Mixture of synchronous and asynchronous approaches.

•Only partly use of the provided tools •Engaging activities - examples of sessions of many hours (4-5 h) in joint activity and discussion

• Innovative use of media and coordination mechanisms

•Good strategies for division of labor•Excellent social dynamics and group spirit.

37/60

A distance learning course of Hellenic Open University (HOU) (2003-2004)

M. Xenos, N. Avouris, D. Stavrinoudis, and M. Margaritis, Introduction of synchronous peer collaboration activities ina distance learning course, IEEE Transactions inEducation, vol. 52 ( 3), Aug. 2009, pp. 305 - 311,

38/60

Synergo server

ODL Server (forum, exchange of material,

help desk) Asynchronous interaction

Synchronous interaction (share

drawing / chat communication)

Synergo client

Synergo client

ODL repository

Activity logging

Submit final solution Record

activity

Student #1 Student #2

Post assignments, form groups

Tutor

Facilitator

Arrangements on sessions plan- direct contact

Respond to technical and organizational problems –

follow activity

GroupGroup

Mixed media and collaboration approachesAsynchronous group activity

Synchronous activity

39/60

Synergo- Discussion forum

40/60

Findings of the HOU study

• Infrastructure overhead higher than expected (unforeseen technical problems)

• Peer tutoring patterns emerged in higher degree than younger students

• Multiple media engaged• Strong social aspects of community

building

41/60

Study on Mecahnics of Collaboration:Coordination protocol

Group B No floor control: all partners can act in the shared work space

Group A Explicit floor control: Only the key owner can act in the shared work space

0

20

40

60

80

100

120

140

160

180

200

Critical Insert Delete Move Chats

Type of events

Num

ber o

f eve

nts

GROUP A (with key)

GROUP Β (without key)

T+ T-

42/60

Findings of the study§ Explicit floor control of the shared activity area did not inhibit problem solving

§ Similar patterns of activity in both groups.

§ group T- was more active than T+

§ T+ students have been obliged to negotiate on possession of the activity enabling key and thus argue at the meta-cognitive level of the activity and externalise their strategies, a fact that helpedthem deepen their collaboration

43/60

models

44/60

#1 Support for Group Awareness through a Machine Learning ApproachTrain a classifier to be used for estimation of the quality of collaboration using historical data of problem solving activities of students engaged in building concept maps and flow-chart diagrams in Hellenic Open University and University of PatrasM. Margaritis, N. Avouris, G. Kahrimanis, On Supporting Users’Reflection during Small Groups Synchronous Collaboration, 12th International Workshop on Groupware, CRIWG 2006 Valladolid, Spain, September 17-21, 2006, LNCS 4154

45/60

Logfile segmentation L={S1, S2, … Sk}

NE

quality of collaboration per segment (bad, average, good)

46/60

Correlation based feature selection(CFS) for different segment sizes

NE=60 NE=80 NE=100 NE=200 (2) num_chat (2) num_chat (2) num_chat (2) num_chat

(3)symmetry_chat (3)symmetry_chat

(4) altern_chat (4) altern_chat (4) altern_chat (4) altern_chat

(5) avg_words (5) avg_words (5) avg_words (5) avg_words

(6) num_quest (6) num_quest (6) num_quest

(7) num_draw (7) num_draw (7) num_draw (7) num_draw Correlation based Feature Selection (CFS)

technique:

makes use of a heuristic algorithm alongwith a gain function to validate theeffectiveness of feature subsets.

NE= number of events per segment

47/60

Performance of classificationalgorithms

• Naïve BayesianNetwork

• Logistic Regression• Bagging• Decision Trees• Nearest Neighbor 75

80

85

90

60 80 100 200Fragmentation factor NE

Suc

cess

rate

(%)

NaiveBayesLogisticBaggingSimpleLogisticRandomForestNNge

48/60

Visualization of group awareness indicator

State of Collaboration

49/60

Evaluation study• 11 groups of 3 students each were given a

collaborative task. • 6 of these groups were provided with the group

awareness mechanibsm. • 5 groups did not have that facility• The mean values of collaboration symmetry

were significanlty different between the two sets (p=0,0423).

50/60

Side-effect

• in four (4) out of the six (6) groups therewas an explicit discussion about the groupawareness mechanism.

• A side-effect:

51/60

#2 Measuring quality of collaboration in Synergoactivities using a rating scheme and an automatic rating model

Based on: Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2, 63–86.

52/60

Original setting New setting

Desktop-videoconferencingsystem with shared texteditor

Synergo: shared whiteboardand chat

Medical decision making Computer programming(algorithm building)

Intermediates;complementary prior

knowledge (psychology andmedicine)

Beginners;similar prior knowledge

CSCL tool

Domain

Collaborators

Meier et al. (2007) rating scheme

53/60

Meier et al (2007) rating scheme dimensions

54/60

Kahrimanis et al. (2009) adapted collaboration rating scheme

7. Individual Task Orientation (for dyad mean or absolute difference)

Motivation

6 .Cooperative Orientation Interpersonal Relationship

5 .Structuring the Problem Solving ProcessCoordination4. Argumentation

3. Knowledge Exchange Joint information processing

2. Sustaining Mutual Understanding 1. Collaboration Flow Communication

Dimensions Aspect of collaboration

55/60

Development of a Collaboration Quality Estimation Model

Data set used• 350 students of 1st year working in

dyads to solve an algorithmic problem using Synergo (academic year 2007-2008) duration of activity 45’ to 75’

• 260 collaborative sessions• Grading according to the quality of

solution and quality of collaboration

56/60

36 derived metrics used(Kahrimanis et al. 2010)

57/60

Quality of Collaboration Estimator(Kahrimanis et al. 2010)

Observed vs. Estimated CQ average

-2

-1

0

1

2

3

-2 -1 0 1 2 3

Estimated(collaboration quality avg)

Obs

erve

d (c

olla

bora

tion

qual

ity a

vg)

VIPs (1 Comp / 95% conf. interval)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Variable

VIP

collaboration_quality_avg = 0.460 + 0.004*C4 - 0.005*C6 + 0.011*C8_17.5 - 0.012*C7

+ 0.602*EV3 + 0.447*STC - 0.001*MW1 + 0.008*MW6

Stone & GeisserCoefficient

(cross validation)

Partial Least Squares Regression Model

58/60

Use of Quality of Collaboration Estimator as discriminator between cases of good and bad collaboration

• The model scored between 76.6% to 79.2%, with the exception of one dimension of lower quality.

59/60

Use of Quality of Collaboration Estimator as automatic rater

• The model had acceptable performance as rater as the inter-rater reliability with human raters had the following values: ICC=.54, Cronbach’s α=.70, Spearman’s ρ=.62 acceptable for α και ρ (George, & Mallery, 2003; Garson, 2009), not for ICC(.7) (Wirtz & Caspar, 2002) . This applies both for the average collaboration quality value and the individual dimensions.

60/60

Current developments• Study of tablet-based collaboration patterns

(synergo v. 5)

• Study of Attention mechanisms (Chounta et al. 2010)

61/60

More on Synergo:hci.ece.upatras.gr/synergo

62/60

Some more key references• Avouris N., Margaritis M., & Komis V. (2004). Modelling interaction

during small-group synchronous problem-solving activities: TheSynergo approach, 2nd Int. Workshop on Designing ComputationalModels of Collaborative Learning Interaction, ITS2004, Maceio, Brasil, September 2004.

• Κahrimanis, G., Meier, A., Chounta, I.A., Voyiatzaki, E., Spada, H., Rummel, N., & Avouris, N. (2009). Assessing collaboration quality insynchronous CSCL problem-solving activities: Adaptation andempirical evaluation of a rating scheme. Lecture Notes in ComputerScience, 5794/2009, 267-272, Berlin: Springer-Verlag.

• Kahrimanis G., Chounta I.A., Avouris N., (2010) Determiningrelations between core dimensions of collaboration quality - A multidimensional scaling approach, In the 2nd InternationalConference on Intelligent Networking and Collaborative Systems(INCoS 2010)