sept. 29 th, 2005 investigating learning over time mingyu feng neil heffernan longitudinal analysis...

Sept. 29th, 2005

Investigating Learning over Time

Mingyu FengNeil Heffernan

Longitudinal Analysis on Assistment Data

Purpose

To answer two type of questions: Do students learn over time?

To characterize each person’s pattern of change over time

Within-individual change over time

Do students differ on learning rate? If yes, what impact learning?

To examine if people differ on within-individual change and ask for the association between predictors and patterns of change

Inter-individual differences in change

Important Features of Study of Change

Three or more waves of data Why three?

Describe process the change Tell the shape of individual growth trajectory Is change steady?

The more the better A sensible metric for time

Make sure you still got enough waves of data An outcome whose values change

systematically over time

Method

Our work follows the approach presented in Applied Longitudinal Data

Analysis (Modeling change and event occurrence), Singer & Willett (2003)

Statistical software

package - SPSS was used to run all the analysis.

Data from the Assistment System

Log data of year 2004 841 students 2 Worcester Public Schools 8 Teachers with 4 from each school

Students went to labs about every other week from Sept., 04 to June,05 On average, 5.70 measurement occasions ranges from 1 to 9

The Three Features in the Data

Longitudinal data - “Person-period” structured Each student has multiple records-one for each

measurement occasion Metric for time

“CenteredMonth” – Month centered around Sept. (value = # of months since Sept. [0 … 10])

multiple sessions in one month are aggregated into one

The outcome % correct on the main question. Transformed to “MCASScore”

MCASScore = % correct * 54 (full score of MCAS test)

Sample Data

2 73 n n 104 y 74 1 950 m n n 2 16.61538

3 73 n n 104 y 74 1 950 m n n 3 20.25

4 73 n n 104 y 74 1 950 m n n 5 41.72727

5 73 n n 104 y 74 1 950 m n n 6 23.14286

6 73 n n 104 y 74 1 950 m n n 7 32.4

7 73 n n 104 y 74 1 950 m n n 8 19.63636

9 73 n n 104 y 74 1 951 f y y 2 18

10 73 n n 104 y 74 1 951 f y y 3 33.42857

11 73 n n 104 y 74 1 951 f y y 5 31.90909

School ID Teacher ID Class ID Class Level

Student ID Gender Free lunch?

Special Ed.Centered

Month

MCASScore

Note: 1. class level was determined class average initial score in Oct. : If avg (class score in Oct.) > global mean, then class_level = 1; else class_level = 0.2. Given the way class level is calculated, we filtered out data waves in Sept. and Oct.

Explore the data set

Mean (MCASScore) increased across time

0 1 2 3 4 5 6 7 8 9

CenteredMonth

0.00

5.00

10.00

15.00

20.00

25.00

30.00

Me

an

MC

AS

Sc

ore

School 73 School 75

Mean24.3295 20.6751

Std. Dev. 13.69403 13.60057

T-test showed that students from School 73 has got significant higher scores (p < .001)

Individual Change over Time

Empirical growth plots for 24 students

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

239 240 243 244 245

246 247 248 314 315

316 320 321 327 331

666 667 668 669 805

806 807 809 810

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0 2 4 6 8

CenteredMonth

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0 2 4 6 8

CenteredMonth

0 2 4 6 8

CenteredMonth

0 2 4 6 8

CenteredMonth


Smooth nonparametric summaries of how individuals change over time

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

239 240 243 244 245

246 247 248 314 315

316 320 321 327 331

666 667 668 669 805

806 807 809 810

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0 2 4 6 8

CenteredMonth

0.00

9.00

18.00

27.00

36.00

45.00

54.00

MC

AS

Sco

re

0 2 4 6 8

CenteredMonth

0 2 4 6 8

CenteredMonth

0 2 4 6 8

CenteredMonth


Fitted trajectories (linear regression line)

Mean (Intercept) = 16.555Mean (Slope) = 1.4586 > 0Correlation (intercept, slope) = -.81

Parameters of regression line for some students

Use Multilevel Model for Change

We need a model that embodies two types of research questions Level-1 question about within-person change Level-2 question about between-person

differences Multilevel statistical model

Level-1 submodel that describes how individuals change over time

Level-2 submodel that describes how these changes varies across individuals

A multilevel model Level-1 submodel (individual growth model):

Level-2 submodel:

Composite model:

ijijiiij TIMEY 10

iii LEVEL 111101 iii LEVEL 001000

InterpretationYij : score for person i at time j, a linear function of TIMEij (CenteredMonth here)

∏0i/ ∏1i : Intercept/Slope of true change trajectory of student i (∏0i :score in Sept.)

r00/r10: Population average of level-1 intercept/slope

r01/r11: Population average difference in ∏0i/ ∏1i for a unit difference in level-2 predictor (LEVELi)

εij: random measurement error for person i at occasion j

ζ0i, ζ1i: parameter residual which permits the level-1 parameters of one person to differ stochastically from those of others

)*(*** 1011100100 ijiiijijiijiij TIMETIMELEVELTIMELEVELY

2110

0120

1

0 ,0

0~

Ni

i

2,0~ Nij

Fixed effect

Random effect (Variance

Components)

Mixed Effect Model

Baseline: Unconditional means model

Just means and variations (no predictor)

Fit Multilevel Model to Assistment Data

ijiijY 0 ii 0000

Estimates of Fixed Effects (a)

Parameter Estimate Std. Error df t Sig.

95% Confidence Interval

Lower Bound Upper Bound

Intercept 24.1687396 .3258244 768.537 74.177 .000 23.5291282 24.8083511a Dependent Variable: MCASScore.

This tells us MCASScore varies over time and students differ from each other on MCASScore (p < .001)– sufficient variation at both levels to warrant further analysis

Estimates of Covariance Parameters(a)

Parameter Estimate Std. Error Wald Z Sig. 95% Confidence Interval


Residual 126.8945794 3.1537978 40.235 .000 120.8613877 133.2289377

Intercept [subject = studentID]

Variance55.8742819 4.2160555 13.253 .000 48.1929604 64.7799047

estimated overall average MCASScore

within-person variancebetween-person

variance

More models

Introduce first predictor: TIME Unconditional growth model

This factor is important: BIC diff = 84

Try more factors as predictors school, teacher, class, class_level

ijijiiij TIMEY 10 ii 0000 ii 1101

Estimates of Fixed Effects(a)

Parameter Estimate Std. Error df t Sig. 95% Confidence Interval


Intercept 20.8622499 .5366397 677.170 38.876 .000 19.8085722 21.9159277

CenteredMonth .6415105 .0948756 630.321 6.762 .000 .4552001 .8278210

estimate of students

average MCASScore in Sep.

Average score increase .64 points

every month

Reject hypothesis of no relationship between

score and TIME

What did We Learn?

School matters? Teacher Class Class level

Fit Multilevel Model to Assistment DataModel Predictors BIC #params

Model A 31711.79 3

Model B CenteredMonth 31627.67 6

Model D CenteredMonth+SchoolID 31616.67 8

Model E CenteredMonth+TeacherID 31671.87 20

Model F CenteredMonth+ClassID 31668.08 70

Model K CenteredMonth+ClassLevel 31457.92 8

Model L CenteredMonth+ClassLevel+SchoolID 31454.602 10

Model M CenteredMonth+ClassLevel+SchoolID (intercept) 31449.059 9

Model N CenteredMonth+ClassLevel+TeacherID 31516.433 22

Model O CenteredMonth+ClassLevel+TeacherID(intercept) 31485.309 15

Model A

Model B

Model D

Model E

Model F

Model K

Model L

Model M

Model N

Model O

31300 31400 31500 31600 31700 31800

Model A

Model D

Model F

Model L

Model N

* BIC of Model M (highlighted in bold) is the lowest among all models A through O. * Predictor: TIME, SCHOOL and Class_Level* School was only used as predictor of intercept (changing rate is not distinguishablebetween schools (p > .05, ns)

Result of Model M Tells

Interpretation Estimate of average initial score of students from lower level classes of school 73 is

17.1389; the score is 14.7419 for students from lower level classes of school 75 From higher level class adds 9 points to average initial score Estimate of change rate of lower level classes is .8173 It seems students from higher level classes learns slower (.3473 points lower every

month) How to use this to calculate student’s score?

Students from 73: Score at Month = 17.1389+9.45*Level + (0.817-0.347*Level)*Month Students from 75: Score at Month = 14.7419+9.45*Level + (0.817-0.347*Level)*Month

Estimates of Fixed Effects (a)



[schoolID=73.0] 17.1389235 .7693093 868.727 22.278 .000 15.6290012 18.6488458

[schoolID=75.0] 14.7419300 .7969344 968.325 18.498 .000 13.1780125 16.3058475

ClassLevel 9.4549425 1.0083900 678.784 9.376 .000 7.4750039 11.4348810

CenteredMonth .8172511 .1385169 755.640 5.900 .000 .5453273 1.0891749

CenteredMonth * ClassLevel

-.3473341 .1899717 650.737 -1.828 .068 -.7203657 .0256974

a Dependent Variable: MCASScore.

A New Data Set Include transfer model information

Does learning rate differ on knowledge components?

Use the basic model: “MCAS5” Outcome: MCASScore for a single standard Time metric: season ( every 3 months)

For more stable estimate of student performance on different knowledge status

Include pretest score from 09/2004 Paper and pencil test given in original format of MCAS ’04

Sample Data (II)

schoolID Teacher ID Class ID studentID Season KC Name MCASScore Pretest

73 104 74 950 0 G-Geometry 30.375 8

73 104 74 950 0 M-Measurement 27 8

73 104 74 950 0 N-Number-Sense-Operations 34.2692 8

73 104 74 950 0 P-Patterns-Relations-Algebra 24.3 8

73 104 74 950 0 D-Data-Analysis-Statistics-Probability 40.5 8

73 104 74 950 1 G-Geometry 43.2 8

73 104 74 950 1 N-Number-Sense-Operations 45 8


73 104 74 950 1 D-Data-Analysis-Statistics-Probability 36 8

73 104 74 950 2 G-Geometry 27 8

73 104 74 950 2 M-Measurement 0 8


For one student: 950

Fit Multilevel Model to Assistment Data (II)

MODEL BIC#param

s Predictors

Model A2 66207.548 3

Model B2 66016.383 6 Season

Model C2 65406.461 10 season + KC name (intercept)

Model C2' 65722.122 10 season + KC name (slope)

Model D2 65287.17 14 season + KC name

Model E2 44588.375 8 season + pretest

Model F2 44580.103 7 season + pretest (intercept)

Model G2 44042.376 15 season + pretest (intercept) + KC name0 20000 40000 60000 80000

Model A

Model B

Model C

Model C'

Model D

Model E

Model F

Model G

TIME is still significant (BIC diff = 191) Knowledge Components as a predictor lead to a big improvement (more

than 700 BIC decrease) Pretest is a even better differentiator (see the big gap between Model E

and Model D)

Why?

Result of Mode G2Estimates of Fixed Effects(a)



[KCName=D-Data-Analysis-Statistics-Probability]

9.2048856 .9340674 1058.357 9.855 .000 7.3720511 11.0377201

[KCName=G-Geometry] 12.6313817 .9237818 1020.453 13.674 .000 10.8186526 14.4441108

[KCName=M-Measurement ] 8.0682218 .9397700 1082.318 8.585 .000 6.2242445 9.9121992

[KCName=N-Number-Sense-Operations ] 18.8854008 .9026647 937.580 20.922 .000 17.1139238 20.6568779

[KCName=P-Patterns-Relations-Algebra ] 19.0073128 .9173294 988.721 20.720 .000 17.2071766 20.8074491

PretestScore .6272369 .0438227 498.598 14.313 .000 .5411371 .7133367

Season ([KCName=D-Data-Analysis-Statistics-Probability])

5.2260148 .5272740 3239.974 9.911 .000 4.1921905 6.2598390

Season ([KCName=G-Geometry ]) -.0723080 .5603555 3488.254 -.129 .897 -1.1709659 1.0263499

Season ([KCName=M-Measurement]) 1.4231592 .6093210 3864.869 2.336 .020 .2285379 2.6177805

Season ([KCName=N-Number-Sense-Operations])

2.5131848 .4729417 2715.492 5.314 .000 1.5858227 3.4405468

Season ([KCName=P-Patterns-Relations-Algebra ])

-1.3867194 .4529750 2490.712 -3.061 .002 -2.2749657 -.4984732

a Dependent Variable: MCASScore.

The estimate of average initial score

on “Patterns” is 19.007, the highest

Pretest score increases by 1, the

estimated initial score increases by .627

estimate of average rate of change on Data-Analysis is

5.226

Negative learning rate indicates “un-learning”

Future work

Try other predictors Gender, class_level, finer grained

transfer models Use fitted model to predict post-test

score or even further real MCAS score

Introduce “Assistment” metrics Performance on “scaffolds”, #hints… Weigh outcome by time spent

Thank you!

Details about this analysis are available at http://www.cs.wpi.edu/~mfeng/analysis

sept. 29 th, 2005 investigating learning over time mingyu feng neil heffernan longitudinal analysis...

Documents