assessments to vam to vas to ees points
DESCRIPTION
Assessments to VAM to VAS to EES Points. July 28, 2014. Which assessments to include?. Science 7 = a + b 1 (Math 6 ) + b 2 (Reading 6 ) + b 3 (Math 5 ) + b 4 (Reading 5 ) + c(Proportion) + e. Who’s in each model?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/1.jpg)
Assessments to VAMto VASto EES Points
July 28, 2014
1
![Page 2: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/2.jpg)
Which assessments to include?Science
Outcome Y 2012 Y = Xt1 + Xt2
2014 2013 2012 2011 2010 2009 2008
11 10 9 8 7
11 10 9 8 7 6
11 10 9 8 7 6 5
7 6 5 4 3
4 3 2 EOY
2
Science7 = a + b1(Math6) + b2(Reading6) + b3(Math5) + b4(Reading5) + c(Proportion) + e
![Page 3: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/3.jpg)
Who’s in each model?
• Models are developed by course group. A teacher is assigned a course group based on the course code of the courses they teach.– A teacher can be in more than one course group (e.g. 5th
grade math and 5th grade reading, or Algebra 1 and Geometry).
– Course groupings help mitigate against bias that may result from an unequal distribution of assessment difficulty and/or student type.
3
![Page 4: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/4.jpg)
Nomenclature• Coding
– t = the current assessment occasion; – t-1 = the prior assessment occasion;– t-2 = the prior assessment occasion to t-1;– SS = Scale score– M Math, R = Reading, Sci = Science.– . denotes class/teacher mean;– .. Denotes the grand mean (usually by course group
• E.G. SSMt = the current scale score in Math for an individual student.
• E.G. SSMt. = the current scale score in Math for an individual student.
4.
![Page 5: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/5.jpg)
Preparing the data
• Step 1:– Normalize the scale scores to a common year
(2012);NSSMt = SSMt – SSM2012../SDM2012
• Where SD = Standard Deviation• N = Normalized.
5
![Page 6: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/6.jpg)
Preparing the data
• Step 2:– Link every student’s current score to the
Conditional standard error of measurement (CSEM).
• Step 3:– Use the Structure table to ensure the proper prior
scores are linked to each student’s current (outcome) score.
6
![Page 7: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/7.jpg)
The Base file (for 2012 7th grade Biology)
7
Each row is a student
Multiple rows will form a teacher’s class.
![Page 8: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/8.jpg)
• Step 4:– The Base file is aggregated by teacher.
8
Each row is a teacher
![Page 9: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/9.jpg)
• Step 5:– This step could be carried out by many different
statistical software applications, but the PED uses HLM.
• HLM has a couple of benefits:– It converges quickly (we ran about 120 VAMs)– Output file efficiently provides necessary results for EES.
9
![Page 10: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/10.jpg)
The basic Model• The outcome variable is NSSSCIT
Summary of the model specified– Level-1 ModelNSSSCITij = β0j + rij
Level-2 Modelβ0j = γ00 + u0j
Mixed ModelNSSSCITij = γ00 + u0j+ rij-> in English = a student’s 7th grade Biology score is a function of the grand mean, of all 7th
grade biology scores, a unique contribution of teachers and a random component.
– This is a mixed effects model.• There are both fixed and random effects.
– Teacher VAS are based on random effects.
– This is the unconditional model.– It is always the first step in VAM modeling.
10
![Page 11: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/11.jpg)
Fixed Effect Coefficient Standarderror t-ratio Approx.
d.f. p-value
For INTRCPT1, β0
INTRCPT2, γ00 -0.004637 0.042111 -0.110 178 0.912
11
Random Effect
StandardDeviation
VarianceComponent d.f. χ2 p-value
INTRCPT1, u0 0.52443 0.27503 178 2757.06740 <0.001
level-1, r 0.89256 0.79667
Final estimation of fixed effects:Final estimation of variance components
![Page 12: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/12.jpg)
• Note: although a “full” model is used to calculate a teacher’s VAS, we will start with the simple model to demonstrate the steps.
• Step 6:• Use HLM results to calculate a teacher’s
unique contribution to student learning (VAS).– Obtain the OLS residual = Observed – expected.
12
![Page 13: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/13.jpg)
OLS ResidOLS = .768 – (-.005) = .773.
13
ObservedOLS Residual
Expected
![Page 14: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/14.jpg)
• Step 7:• Consider the reliability of each teacher’s
estimate reliability = variance of true scores
variance of observed scoresl = t00/(t00+s2/nj)
• Calculate the Empirical Bayes (EB) estimate using the Kelley equation.
14
![Page 15: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/15.jpg)
Reliability of Estimates
• Reliability depends on the degree to which the true underling parameters vary among groups (e.g. schools).
• Classical test theory notion is that reliability = variance of true scores variance of observed scores
l = t00/(t00+s2/nj)
![Page 16: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/16.jpg)
• Step 7 continued– The Kelley equation:bEB = bols(l) + Y(1- l)
16
![Page 17: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/17.jpg)
ResidEB = .768(.97) + -.005(1-.97) = .751.
17
ReliabilityEB Residual
OLS Residual
![Page 18: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/18.jpg)
EB Residual
|residualols| > |residualEB|; |.773| > |.751| ,
Which is why this is termed a “shrunken estimate.”
The EB residual is a teacher’s VAS.
18
![Page 19: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/19.jpg)
• Step 8:• We normalize VAS scores so that results from
all course groups (and assessment types, e.g. EoC, Dibels, etc) will be on the same scale.
• VASnormalized = (VAS –VAS..)/SDVAS
VAS.. is calculated for each Course group.• And where applicable, by course group by grade.
• E.g. VASnormalized = .751 – (-.005)/.4896 = 1.54.
19
![Page 20: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/20.jpg)
• This Teacher’s VAS of 1.54 places him/her in the Highly effective range.
20
![Page 21: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/21.jpg)
• Step 9:– Converting VAS scores into EES points.– Given the normalization in the previous step, we
take the normal CDF of the VAS:
– In excel this is =NORMSDIST(VAS).– And results in:
21
![Page 22: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/22.jpg)
VAS to Points Conversion
22
-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.000
10
20
30
40
50
60
70
0.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0
65.8
0.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0
Value Added Score (VAS)
Poin
ts
VAS = 1.54
![Page 23: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/23.jpg)
• Notes:– The differences between an actual VAS calculation
and the example:• Prior achievement (etc) is included in the student level
model.• Peer effects are included (e.g. class average prior math
and reading achievement).– The level 2 (teacher level) model determines what the EB estimates
will be shrunk towards (in the previous example this was the grand mean because there were no level 2 predictor variables, but for the EES, it includes peer effects).
23
![Page 24: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/24.jpg)
24
![Page 25: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/25.jpg)
• Notes continued:– The actual VAM utilizes the CSEM to eliminate potential
relationships between the predictors and the VAS, as well as to help guard against the impact of outliers (extreme test scores).
• A teacher’s VAS in the Summative Report is the weighted Average of all the available VAS scores for a teacher.– The weights are the number of students that contributed
to a VAS score (which may not equal enrollment ).– This can consist of multiple VAS scores per year and
multiple years.
25
![Page 26: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/26.jpg)
26
E.G. in TotVAS11 = .(58*18+ .93*18)/36 = .76
![Page 27: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/27.jpg)
27
TotVAS_all = value in Summative report and used to calculate points =.76*36 + 1.26*48 + 1.04*19 = 107.6/103 = 1.04.
![Page 28: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/28.jpg)
• VAS score for teacher with unconditional VAS of .751 is– .170 using full model and is – 1.18 when normalized.– This = 61.9 points assuming 70 points possible in
STAM 1.
28
![Page 29: Assessments to VAM to VAS to EES Points](https://reader035.vdocument.in/reader035/viewer/2022062520/56815aa7550346895dc83e89/html5/thumbnails/29.jpg)
How about Excel?Teacher Student Course ID (4) Year
Proportion Grade SSSCIt SSMt-1 SSMt-2 SSRt-1 SSRt-2
1 1 1707 2012 100 7 41 33 29 36 341 2 1707 2012 100 7 33 41 39 61 471 3 1707 2012 100 7 49 51 43 51 461 4 1707 2012 100 7 55 43 45 44 441 5 1707 2012 100 7 51 51 44 48 371 6 1707 2012 100 7 41 44 43 43 471 7 1707 2012 100 7 28 33 38 45 431 8 1707 2012 100 7 26 33 26 29 201 9 1707 2012 100 7 29 31 35 42 431 10 1707 2012 100 7 43 38 38 39 321 11 1707 2012 100 7 48 42 45 47 431 12 1707 2012 100 7 45 42 32 36 281 88 1707 2012 100 7 36 31 37 39 39
Average 45.07 43.20 41.61 47.16 42.43Average of prior Averages 43.60
Estimated VAS 1.63
Estimated Pct of Points Earned 0.95Estimated EES Points (out of 70) 66.37
29
There is no guarantee that this method will provide a close approximation of the actual VAS score – however, the sign and magnitude should provide some approximation.
A regression for each teacher will result in a VAS of 0.