data analysis of coded chats tracking the collaboration activity of a student in different powwows...

Data Analysis of Coded Chats Tracking the collaboration activity of a student in different powwows

Progress Report, VMT Meeting, Feb. 2nd, 2005

Fatos XhafaVMT Project

February 2nd, 2005. VMT Meeting

Outline

The data considered for this analysis

The research questions addressed

The statistical results Discussion Next steps


The data considered for this analysis Student: AME

Participation in 11 powwows (pow9 ― pow19) Only seven of these powwows (pow9, pow10,

pow12, pow13, pow14, pow15, pow18 ) are coded

Powwows considered: pow9, pow10, pow13, pow14, pow18 The descriptive statistics is required as a previous

step Only three of these (pow9, pow10, pow18) are

part of the initial sample of six powwows


The research questions addressed (I) Q1: Is there any significant difference between the AME's “production” and his groups’ “production”?

Q2: In case of significant difference (from Q1), is there any pattern in the variation (difference) between the AME's and his groups’ production?

Q3: Is there any clustering of the AME's “behavior” – as regards its production– in different powwows?

Q4: What are the correlations between the variables for the AME's “production”? Are those different from what we already know from the

correlations for the sample of six powwows?


The research questions addressed (II) Q5: What are the correlations between the

variables for the AME's “production” and the variables of his groups’ “production”?

Q6: How is the AME’s participation chronologically? Is there any evidence of “experience” due to

repeated participation? Or, any other kind of significant influence of

the repeated participation?


The study of AME's “production” vs. his groups’ “production”

(Social reference, Pbm Solving and Math Move vars are used)


The AME's “production” vs. his groups’ “production” There is a significant difference between the

Percentage of Group's Pbm Solving postings and Percentage of AME's Pbm Solving postings

There is a significant difference between the Percentage of Group's Math Move postings and the Percentage of AME's Math Move postings

There is no significant difference (at 95% confidence level) between the Percentage of Group's Social reference postings and Percentage of AME's Social reference postings but a significant difference at 90% confidence level


Analytically…

Mean N Std. Deviation Std. Error Mean

Pair 1

Percentage of Group's Social reference postings

24.3800 5 7.32236 3.27466

Percentage of AME's Social reference postings

35.9200 5 8.64390 3.86567

Pair 2

Percentage Group's Pbm Solving postings

26.3400 5 6.76705 3.02632

Percentage of AME's Pbm Solving postings

34.0400 5 6.62442 2.96253

Pair 3 Percentage Group's Math Move postings

12.6400 5 6.26123 2.80011

Percentage of AME's Math Move postings

31.1200 5 9.29500 4.15685

Paired Samples Statistics


Analytically…

Paired Differences t dfSig.

(2-tailed)

MeanStd.

DeviationStd. Error Mean

95% ConfidenceInterval of theDifference

Lower Upper

Percentage of Group's Social reference postings - Percentage of AME's Social reference postings

-11.54000 11.52749 5.15525 -25.85327 2.77327 -2.238 4 .089

Percentage Group's Pbm Solving postings - Percentage of AME's Pbm Solving postings

-7.70000 4.37893 1.95832 -13.13716 -2.26284 -3.932 4 .017

Percentage Group's Math Move postings - Percentage of AME's Math Move postings

-18.48000 4.20678 1.88133 -23.70341 -13.25659 -9.823 4 .001


Clustering of the AME's “production” in different powwows


Clustering of the AME's “production” in different powwows The study shows the following clustering

(based on the AME’s “production” in the four dimensions, now including Conversation):

pow9, pow13, pow18 pow10, pow14


Analytically…Cluster Membership

Case NumberPowwow

group Cluster

1 pow9 1

2 pow10 2

3 pow18 1

4 pow13 1

5 pow14 2

Final Cluster Centers

Cluster

1 2

Percentage of AME's Conversation postings

29.57 37.35


30.00 44.80


30.40 39.50


26.27 38.40

Distances between Final Cluster Centers

Cluster 1 2

1 22.575

2 22.575

Number of Cases in each Cluster

Cluster 1 3.000

2 2.000

Valid 5.000

Missing .000


Study of correlations between the variables of the AME's “production”


The correlations between the variables for the AME's “production”Correlations

Percentage of AME's

Conversation postings

Percentage of AME's Social

reference postings

Percentage of AME's Pbm

Solving postings

Percentage of AME's Math Move

postings


Pearson Correlation1 .918(*) .827 .782

Sig. (2-tailed) . .028 .084 .118

N 5 5 5 5


Pearson Correlation .918(*) 1 .821 .858

Sig. (2-tailed) .028 . .088 .063

N 5 5 5 5


Pearson Correlation.827 .821 1 .919(*)

Sig. (2-tailed) .084 .088 . .027

N 5 5 5 5


Pearson Correlation.782 .858 .919(*) 1

Sig. (2-tailed) .118 .063 .027 .

N 5 5 5 5* Correlation is significant at the 0.05 level (2-tailed).

No significant at 0.05 level


The correlations between the variables for the AME's “production” – again, Pow18 is an outlier for Social reference dimension!

Correlations (after removing Pow18)

Percentage of AME's Social

reference postingsPercentage of AME's Pbm

Solving postingsPercentage of AME's Math Move postings


Pearson Correlation

1 .987(*) .954(*)

Sig. (2-tailed) . .013 .046

N 4 4 4


Pearson Correlation

.987(*) 1 .926

Sig. (2-tailed) .013 . .074

N 4 4 4


Pearson Correlation

.954(*) .926 1

Sig. (2-tailed) .046 .074 .

N4 4 4

* Correlation is significant at the 0.05 level (2-tailed).


Regressions (AME’s Soc. Ref vs. AME’s Conversation) graphically:

Linear Regression

25.00 30.00 35.00 40.00


25.00

30.00

35.00

40.00

45.00

Per

cen

tag

e o

f AM

E's

So

cial

ref

eren

ce p

ost

ing

s

pow9

pow10

pow13

pow14

pow18

Percentage of AME's Social reference postings = -12.85 + 1.49 * PConAMER-Square = 0.84


Regressions (AME’s Soc. Ref vs. AME’s Conversation) analytically:Model Summary

Model RR

SquareAdjusted R

Square

Std. Error of the

Estimate Change Statistics

R Square Change

F Chang

e df1df2

Sig. F Change

1 .918(a) .843 .790 2.43614 .843 16.055 1 3 .028

a Predictors: (Constant), Percentage of AME's Social reference postingsb Dependent Variable: Percentage of AME's Conversation postings

Model Sum of

Squares df Mean Square F Sig.

1 Regression 95.284 1 95.284 16.055 .028(a)

Residual 17.804 3 5.935

Total 113.088 4

a Predictors: (Constant), Percentage of AME's Social reference postingsb Dependent Variable: Percentage of AME's Conversation postings

ANOVA(b)


Regressions (AME’s Pbm Solving vs. AME’s Math Move) graphically:

Linear Regression

28.00 32.00 36.00 40.00


20.00

30.00

40.00

Per

can

tag

e o

f AM

E's

Mat

h M

ove

po

stin

gs

pow9

pow10

pow13

pow14

pow18

Percantage of AME's Math Move postings = -12.79 + 1.29 * PPbmSAMER-Square = 0.85


Regressions (AME’s Pbm Solving vs. AME’s Math Move) analytically:

Model Summary

Model RR

SquareAdjusted R Square

Std. Error of the

Estimate Change Statistics

R Square Change

F Change df1 df2

Sig. F Change

1.919(a) .845 .794 4.22240 .845 16.384 1 3 .027

a Predictors: (Constant), Percentage of AME's Pbm Solving postingsb Dependent Variable: Percentage of AME's Math Move postings

Model Sum of

Squares df Mean Square F Sig.

1 Regression 292.102 1 292.102 16.384 .027(a)

Residual 53.486 3 17.829

Total 345.588 4

a Predictors: (Constant), Percentage of AME's Pbm Solving postingsb Dependent Variable: Percentage of AME's Math Move postings

ANOVA


The regression between the AME's variables Soc. Ref, Pbm Solving and Math Move (after removing Pow18)

30.00 35.00 40.00 45.00


20.00

30.00

40.00

Per

can

tag

e o

f AM

E's

Mat

h M

ove

po

stin

gs

pow9

pow10

pow13

pow14

Percantage of AME's Math Move postings = -9.18 + 1.07 * PSRefAMER-Square = 0.91

30.00 35.00 40.00 45.00


28.00

32.00

36.00

40.00

Per

cen

tag

e o

f AM

E's

Pb

m S

olv

ing

po

stin

gs

pow9

pow10

pow13

pow14pow18

Percentage of AME's Pbm Solving postings = 11.44 + 0.63 * PSRefAMER-Square = 0.67


Study of correlations between the AME's vars and his groups’ vars


Correlation between AME’s and his Groups’ vars (Social Reference)

LLR Smoother

15.00 20.00 25.00 30.00


30.00

40.00

50.00

60.00

70.00

80.00

Per

cen

tag

e o

f AM

E's

So

cial

ref

eren

ce p

ost

ing

s

pow9

pow10

pow13

pow14

pow18

No correlation between AME’s Soc. Reference and Groups’

Soc. Reference!


Correlation between AME’s and his Groups’ Math Move (significant correlation)Correlations

Percentage Group's Math Move postings

Percentage of AME's Math Move

postings


Pearson Correlation 1

.927(*)

Sig. (2-tailed).

.023

N 5 5

Percantage of AME's Math Move postings

Pearson Correlation .927(*)

1

Sig. (2-tailed).023

.

N 5 5

* Correlation is significant at the 0.05 level (2-tailed).

Linear Regression

20.00 30.00 40.00


5.00

10.00

15.00

20.00

Per

cen

tag

e G

rou

p's

Mat

h M

ov

e p

ost

ing

s

pow9

pow10

pow13

pow14

pow18

Percentage Group's Math Move postings = -6.79 + 0.62 * PMathAMER-Square = 0.86


Correlations between AME’s and his Groups’ Pbm Solving (almost significant, 0.057 level)

20.00 24.00 28.00 32.00


28.00

32.00

36.00

40.00

Per

cen

tag

e o

f AM

E's

Pb

m S

olv

ing

po

stin

gs

pow9

pow10

pow13

pow14pow18

Percentage of AME's Pbm Solving postings = 13.76 + 0.77 * PPbmSoGrR-Square = 0.62

Percentage of AME's Pbm

Solving postings

Percentage Group's Pbm

Solving postings

Pearson Correlation


1.000 .786


.786 1.000

Sig. (1-tailed) Percentage of AME's Pbm Solving postings

. .057


.057 .

N Percentage of AME's Pbm Solving postings

5 5


5 5


Study of AME’s participation chronologically[“Who shapes the collaboration?”]


AME’s participation chronologically (Pbm Solv.)

04/15/200404/25/2004

05/06/200405/09/2004

05/27/2004

Date of the powwow

15.00

20.00

25.00

30.00

35.00

40.00

45.00 Percentage of AME's Pbm Solving postings


High “agreement” in the shape of the two

curves!!

[Not surprising, as we know the correlations between the AME’s vs. his groups’ vars]


AME’s participation chronologically (Math Move)

04/15/200404/25/2004

05/06/200405/09/2004

05/27/2004

Date of the powwow

10.00

20.00

30.00

40.00



High “agreement” in the shape of the two

curves!!



AME’s participation chronologically (Soc. Ref)

04/15/200404/25/2004

05/06/200405/09/2004

05/27/2004

Date of the powwow

10.00

20.00

30.00

40.00

50.00 Percentage of AME's Social reference postings


No “agreement” in the shape of the two

curves!!



Discussion…back to the research questions Q1: Is there any significant difference between

the AME's “production” and his groups’ “production”? Yes, as regards the Pbm Solving and Math Move. No, as regards the Social reference, though there is a

considerable difference (significant at 90% conf. level). Q2: In case of significant difference (from Q1), is

there any pattern in the variation between the AME's and the groups? Yes, the variation follows a pattern-magnitude difference

(agreement curves). The AME’s “production” is always above the groups’

“production”, in the three dimensions! This might be indicating that AME is a “high ability” student.


An excerpt from AME’s “production” (Pow10, Conversation and Math Move dimensions)

AMEFIRMCPMFP

Handle

Pies show percents

35.48%

22.58%

41.94%

Agree Critique Disagree Explain

Elaborate Extend Follow No Code

Offer Regulate Respond Request

Repair typing State Setup

61.54%30.77%

7.69%

66.67%

33.33%46.67%

6.67%

40.00%

6.67%

38.46%

61.54%

29.17%

6.25%56.25%

8.33%

26.19%

61.90%

11.90%

100.00%

47.22%

1.39%

51.39%63.64%

9.09%

27.27% 33.63%

19.47%24.78%

22.12%39.02%

13.41%

34.15%

13.41%

44.44%

33.33%

22.22%

45.07%

12.68%

25.35%

16.90%

66.67%

20.00%

13.33%

Pie chart of Conversation dimension sliced by Handle

AMEFIRMCPMFP

Handle

Pies show percents100.00%

Algebraic expr. Geometric expr. Import new math

Import & apply new math Numeric computation

36.14%

1.20%

57.83%

4.82%

80.00%

20.00%

57.14%

42.86%50.00%

6.67%

43.33%

Pie chart of Math Move dimension sliced by Handle


Discussion…back to the research questions Q3: Is there any clustering of the AME's

“behavior” – as regards its production– in different powwows? Yes, the data shows a clustering into two groups

of powwows (pow9, pow13, pow18 and pow10, pow14)

We know that pow18 and pow10 fall into different clusters, is the same reason for AME’s clustering?

What is what makes the AME’s behavior to be clustering in this way?


Discussion…back to the research questions Q4: What are the correlations between the

variables for the AME's “production”? Are those different from what we already know from the correlations (the sample of six powwows)? We have now a new correlation, between Conversation

and Social reference, namely a strongly positive correlation, .918)

The correlation between the Pbm Solving and Math Move is as in the case of the sample (strongly positive correlation, .919)

The correlation between Social Ref. and Pbm Solving and Social Ref. and Math Move is reversed (positively correlated)!!

Why? How does the AME’s Soc. Ref activity unfolds in way that it is positively correlated to the Pbm Solving and Math Move?


Why positive correlation between Social Ref. & Pbm Solving and Social Ref. & Math Move for AME?

SOCIAL REFERENCEPOSTINGS

OFF TOPIC POSTINGS(POSTINGS NOT CODED

IN THE PBM SOLVING DIMENSION)

POSTINGS CODED IN THE PBM SOLVING

DIMENSION

These postings (could) negatively

influence the production of pbm

solving

These postings DO NOT influence negatively the

production of pbm solving


This leads to…

Hypothesis:

More “off topic” postings in the social

reference implies more negative influence

of social reference in the production

of pbm solving


Why positive correlation between Social Ref. & Pbm Solving and Social Ref. & Math Move for AME? (POWWOW10)

SOCIAL REFERENCEPOSTINGS(166 overall)

OFF TOPIC POSTINGS(POSTINGS NOT CODED

IN THE PBM SOLVING DIMENSION)

POSTINGS CODED IN THE PBM SOLVING

DIMENSION

AME: 13/81=16%

AME: 68/81=84%

FIR: 14/34=42%

MCP: 8/51=15%

FIR: 20/34=58%

MCP: 43/81=85%


Why positive correlation between Social Ref. & Pbm Solving and Social Ref. & Math Move

for AME?

AMEFIRMCP

Handle

Pies show percents

60.47%

2.33%

37.21%

Collaboration group Collaboration individual Greet Home

Interest Identify other Identify self Risk-taking

School Sustain climate

44.12%

20.59%

35.29%50.00%

25.00%

25.00%

100.00%

83.33%

16.67%

50.00%50.00%

75.00%

25.00% 25.00%

75.00%

30.77%

30.77%

38.46%

58.33%25.00%

16.67%

Pie chart of Social reference sliced by Handle

Most of the “Identify self” are from AME”

(showing his own abilities/math skills)

Most of the “Group refs” are by AME”(addressing the

group)

Most of the “Group refs” are by AME”(addressing the

group)


Discussion…back to the research questions Q5: What are the correlations between the

variables for the AME's “production” and the variables of his groups’ “production”? There is No correlation between AME’s Soc.

Reference and Groups’ Soc. Reference! There is a significant correlation between AME’s

and his Groups’ Math Move There is (almost) a significant correlation between

AME’s and his Groups’ Pbm Solving


Discussion…back to the research questions Q6: How is the AME’s participation chronologically? Is

there any evidence of “experience” due to repeated participation? Or any kind of significant influence of the repeated participation? From the analysis we see that repeated participation doesn’t

mean that the participant’s production is greater and greater… It could be indicating that “no matter how experienced or high-

ability you are, your production is by large a matter of the group”!!

How does the group’s “circumstances/conditions” influence the participants’ production (even in the sense of “producing” more or less?)

Yet, an experienced participant can show/keep a pattern of “producing” more than the group does! This is the case of AME


Next steps From statistical perspective

Complete the same computations by including the 2 powwows (and later with the rest of 4 powwows that are not coded yet)

Refine the analysis (second level analysis including subcategories of the dimensions)

Consider another student (FIRSUNSHINE?) that could be another interesting case (“low-ability”?) preferably from the AME’s powwows

From thread-based analysis Analyze the AME’s participation by looking at the “patterns” produced by

AME Analyze the AME’s participation by looking at the threads in which he is

involved From interaction based / CA perspective

Analyze how does the AME’s activity unfold sequentially in different powwows, especially his social reference production

How does the group’s “circumstances/conditions” influence the participants’ production (even in the sense of “producing” more or less?)

How does the type of organization (exposition vs. exploration) influence the AME’s production?

How can, from a CA perspective, be explained the AME’s clustering?

data analysis of coded chats tracking the collaboration activity of a student in different powwows...

Documents