william h. angoff memorial lecture inferences about teachers based on student … · 2016. 5....
TRANSCRIPT
![Page 1: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/1.jpg)
1
Inferences About Teachers Based on Student Test Scores
Edward Haertel Stanford University
William H. Angoff Memorial Lecture
Washington, DC March 22, 2013
![Page 2: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/2.jpg)
2013 William H. Angoff Memorial Lecture
Policy Context Sense of urgency about reform ◦ Persistent achievement gaps ◦ International rankings ◦ 21st century skills
Evidence of powerful teacher effects Weak, ineffective teacher evaluation Finding and removing ineffective
teachers as a potent path for education policy?
2
![Page 3: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/3.jpg)
2013 William H. Angoff Memorial Lecture
The Allure of VAM Value-Added Models (VAM) use student
test score gains to provide direct evidence of teacher effectiveness ◦ Data linking student test scores to teachers
are much improved ◦ Statistical models seem to promise
reliable and valid results ◦ Extraordinary benefits are projected
3
![Page 4: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/4.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
4
![Page 5: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/5.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
5
![Page 6: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/6.jpg)
2013 William H. Angoff Memorial Lecture
How Big are Teacher Effects?
6
Teacher
Other School Factors
Out-of-School Factors
Unexplained Variation
Influences on Student Test Scores
![Page 7: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/7.jpg)
2013 William H. Angoff Memorial Lecture
How Big are Teacher Effects?
7
OUT OF REACH?
Teacher
Other School Factors
Out of reach? Out of reach?
BIG, as potential policy variable
Influences on Student Test Scores
![Page 8: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/8.jpg)
2013 William H. Angoff Memorial Lecture
How Big are Teacher Effects?
8
Teacher
Other Factors
Random noise plus systematic bias
SMALL, as “signal” relative to “noise”
Influences on Student Test Scores
![Page 9: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/9.jpg)
2013 William H. Angoff Memorial Lecture
Problems With the Logic Cannot reliably identify top-quintile teachers Effects fade over time “Top-quintile” teachers are in short supply Proposal to replace “worst” with
average teachers is similarly flawed
9
![Page 10: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/10.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
10
![Page 11: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/11.jpg)
2013 William H. Angoff Memorial Lecture
A Close Look at Test Score Gains
11
![Page 12: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/12.jpg)
2013 William H. Angoff Memorial Lecture
Consequences of Nonlinear Scale
A nonlinear scale means teachers are rewarded or penalized, depending on where their students start out ◦ Especially problematical for teachers of students
above or below grade level, or with special needs
12
Measured Growth = 6 points
Measured Growth = 7 1/2 points
![Page 13: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/13.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
13
![Page 14: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/14.jpg)
2013 William H. Angoff Memorial Lecture
Comparison to a Familiar Testing Situation
14
Typical Test Simplified Teacher VAM
Examinees Students Teachers
Items Test questions Students
Test Items in a test form Students in a classroom
Administration Student answers items Teacher teaches students
Item Scoring Item responses scored according to key
Student learning “scored” by giving each student a standardized test
Test Score Sum of Item scores
Average of student test scores
This simplified version cannot work, because teachers get “tests” (classes) of varying difficulties (prior knowledge, educational challenge)
![Page 15: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/15.jpg)
2013 William H. Angoff Memorial Lecture
Accounting for Student Differences Begin with average score for all students Make Adjustments ◦ Remove whatever teacher is not responsible for ◦ Leave whatever teacher is responsible for
Assume that everything left over (i.e., not explained or accounted for) is: ◦ “Effect” of particular teacher and/or ◦ Random (or nonsystematic) variation
15
In other words, we assume that once adjustments are made, assignments of students to teachers may be regarded as random.
![Page 16: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/16.jpg)
2013 William H. Angoff Memorial Lecture
What to Adjust For? Prior-year test scores Test scores from two or more years earlier Absences, suspensions, grade retentions “English learner” or “special education” status Title I eligibility Student mobility Summer school attendance Gender …
16
![Page 17: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/17.jpg)
2013 William H. Angoff Memorial Lecture
Los Angeles Value-Added Model
17
“Stripped of the Greek symbols and statistical jargon, [the] Los Angeles Value-Added Model … , in essence, claims that once we take into account five pieces of information about a student, the student’s assignment to any teacher in any grade and year can be regarded as occurring at random.”
(Briggs & Domangue, 2011, p. 4)
![Page 18: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/18.jpg)
2013 William H. Angoff Memorial Lecture
Background Variables in the LA Value Added Model Prior-year test scores Gender English language proficiency Eligibility for Title I Whether student entered LAUSD schools
after kindergarten
18
![Page 19: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/19.jpg)
2013 William H. Angoff Memorial Lecture
Besides Student Adjustments… Available instructional materials, resources Classroom aides Other teachers Student peers School safety, climate, policies Out-of-school influences during the year …
19
![Page 20: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/20.jpg)
2013 William H. Angoff Memorial Lecture
Estimating Unobserved Scores How well can VAM models adjust for student
and school differences?
20
There is essentially no “mixing” of students from most affluent versus least affluent schools. While students do change teachers and schools, they rarely make large moves up or down across social strata.
“Given the reality of school segregation on the basis of various demographic characteristics of students, including family socioeconomic background, ethnicity, linguistic background, and prior achievement … in practice, some students [may] have no access to certain schools.” Reardon & Raudenbush (2009, p. 494)
![Page 21: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/21.jpg)
2013 William H. Angoff Memorial Lecture
Peer Effects Due To… Group work Peer culture Collective influence on pacing Classroom “chemistry”
21
![Page 22: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/22.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
22
![Page 23: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/23.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument Scoring ◦ Observed score meaning
(bias? other evidence of effectiveness?)
Generalization ◦ From observed score to universe score
(stability/consistency? reliability?)
Extrapolation ◦ From universe score to target score
(target qualities fully captured?)
Implication ◦ From target score to specific interpretations or decisions
(usefulness for specific purposes? unintended effects?)
23
![Page 24: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/24.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument
24
1. Scoring 3. Extrapolation
2. Generalization 4. Implication
This class, this year, this test
Bias?
Other classes, other years
Random Error?
Other tests, nontest outcomes Other Measures?
Validity of possible uses
Consequences?
![Page 25: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/25.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument
25
1. Scoring 3. Extrapolation
2. Generalization 4. Implication
This class, this year, this test
Bias?
Other classes, other years
Random Error?
Other tests, nontest outcomes Other Measures?
Validity of possible uses
Consequences?
![Page 26: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/26.jpg)
2013 William H. Angoff Memorial Lecture
Scoring Do teachers’ VAM scores accurately reflect
their actual effectiveness, this year with these students, in teaching the content covered on this test? ◦ Is bias acceptably small? ◦ Do scores reflect quality of teaching versus
characteristics of students and schools?
26
![Page 27: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/27.jpg)
2013 William H. Angoff Memorial Lecture
Bias vs. Noise
+/- equally likely for any teacher
Tends to average out over time or across classes
Example: random variations across classes
+ more likely for some teachers, - for others
Cannot be reduced by averaging over more observations
Several plausible examples
27
Random Error (Noise) (Generalization concern)
Systematic Error (Bias) (Scoring concern)
![Page 28: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/28.jpg)
2013 William H. Angoff Memorial Lecture
Falsification Test
Logically, future teachers cannot influence past achievement
Thus, if a model predicts significant effects of current-year teachers on prior-year test scores, then it is flawed or based on flawed assumptions
28
![Page 29: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/29.jpg)
2013 William H. Angoff Memorial Lecture
Falsification Test Findings Using each of three different VAM
specifications, Rothstein (2010) found large “effects” of students’ fifth grade teachers on their fourth grade test score gains
29
![Page 30: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/30.jpg)
2013 William H. Angoff Memorial Lecture
Falsification Test Findings Briggs & Domingue (2011) applied
Rothstein’s test to LAUSD teacher data analyzed by Richard Buddin for the LA Times ◦ For Reading, ‘effects’ from next year’s teachers
were about the same as from this year’s teachers ◦ For Math, ‘effects’ from next year’s teachers were
about 2/3 to 3/4 as large as from this year’s teachers
30
![Page 31: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/31.jpg)
2013 William H. Angoff Memorial Lecture
Possible Sources of Bias Massively nonrandom assignment of
students to teachers Teachers with special qualifications work
with students with particular needs Nonrandom assignment of teachers to
schools Peer effects not adequately accounted for Tests insensitive to growth of very low-
performing or very high-performing students Differential summer learning loss
31
![Page 32: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/32.jpg)
2013 William H. Angoff Memorial Lecture
Summer Learning Loss
Low-income families: ◦ Summer learning loss ◦ Spring-to-spring gain understates school year gain
High-income families: ◦ Summer learning gain in reading ◦ Spring-to-spring gain overstates school year gain
32
Measured Spring-to-Spring test score gain
Spring-to-Fall (summer) loss or gain
Fall-to-Spring (school year) gain
= +
![Page 33: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/33.jpg)
2013 William H. Angoff Memorial Lecture
Scoring Conclusion VAM scores do capture meaningful
differences in teaching effectiveness, but … There is compelling evidence of systematic
bias in teacher VAM estimates ◦ Falsification test findings ◦ Documented patterns of summer learning loss
VAM scores reward or penalize teachers not only for how well they teach, but also for whom they teach and where they teach 33
![Page 34: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/34.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument
34
1. Scoring 3. Extrapolation
2. Generalization 4. Implication
This class, this year, this test
Bias?
Other classes, other years
Random Error?
Other tests, nontest outcomes Other Measures?
Validity of possible uses
Consequences?
![Page 35: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/35.jpg)
2013 William H. Angoff Memorial Lecture
Reliability Describes score stability or consistency The correlation between two measurements
indicates reliability ◦ reliability (correlation) coefficient goes from zero
(no linear relation) to one (perfect linear relation)
35
r = .00 r = .50 r = .80 r = .90 r = 1.00
![Page 36: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/36.jpg)
2013 William H. Angoff Memorial Lecture
Year-to-Year Changes in Ranks
36
Based on McCaffrey, Lockwood, Sass, & Mihaly, 2009, Table 4 (p. 591)
Next-Year Distribution of One Year’s Bottom-Quintile Elementary Teachers, in Five Florida Counties
Next-Year Distribution of One Year’s Top-Quintile Elementary Teachers, in Five Florida Counties
0 5
10 15 20 25 30 35 40 45
Bottom Quintile
2 3 4 Top Quintile
Perc
ent Dade
Duval
Hillsborough
Orange
Palm Beach
0 5
10 15 20 25 30 35 40 45
Bottom Quintile
2 3 4 Top Quintile
Perc
ent Dade
Duval
Hillsborough
Orange
Palm Beach
![Page 37: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/37.jpg)
2013 William H. Angoff Memorial Lecture
Effectiveness Varies Year-to-Year
Does each teacher have an “effectiveness” that is constant over time?
37
“Approximately one-third to one-half of the variation in teacher effects is simply due to sampling error or noise in student achievement. Of the remaining variance, between one-third and two-thirds is attributable to variation in effectiveness within teachers over time.”
McCaffrey, Sass, Lockwood, & Mihaly, 2009, p. 599
![Page 38: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/38.jpg)
2013 William H. Angoff Memorial Lecture
Reliabilities from MET Project
38
Same Year, Different Course
Sections Different
Years
State Math Test 0.381 0.404
State English Language Arts Test 0.180 0.195
Balanced Assessment in Mathematics
0.228
Stanford 9 Open-Ended Reading 0.348
Findings from Measures of Effective Teaching (MET) Project, Bill & Melinda Gates Foundation, 2010, Tables 6, 7, and 8
![Page 39: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/39.jpg)
2013 William H. Angoff Memorial Lecture
Averaging Over 2-3 Years Helps
39
Reliability of Single-Year Data
Reliability of 2-Year Average
Reliability of 3-Year Average
.20 .33 .43
.30 .46 .56
.40 .57 .67
.50 .67 .75
r = .40 r = .57 r = .67
![Page 40: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/40.jpg)
2013 William H. Angoff Memorial Lecture
From a Teacher in Houston, TX
40
“I do what I do every year. I teach the way I teach every year. [My] first year got me pats on the back; [my] second year got me kicked in the backside. And for year three, my scores were off the charts. I got a huge bonus, and now I am in the top quartile of all the English teachers. What did I do differently? I have no clue.”
Darling-Hammond, Amrein-Beardsley, Haertel, & Rothstein (2012, p. 11.)
![Page 41: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/41.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument
41
1. Scoring 3. Extrapolation
2. Generalization 4. Implication
This class, this year, this test
Bias?
Other classes, other years
Random Error?
Other tests, nontest outcomes Other Measures?
Validity of possible uses
Consequences?
![Page 42: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/42.jpg)
2013 William H. Angoff Memorial Lecture
Extrapolation How do teacher VAM scores correlate with
other indicators of teaching quality? How much do rankings change if a different
student achievement test is used? Does achievement test content capture
valued learning outcomes? How do VAM scores relate to valued non-
test (non-cognitive) outcomes?
42
![Page 43: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/43.jpg)
2013 William H. Angoff Memorial Lecture
VAM Scores vs. Other Evidence
43
Teacher Description Based on Classroom Observations
“She reasons incorrectly about unit rates. She concludes that an answer of 0.28 minutes must actually be 0.28 seconds …. She tells students that integers include fractions. She reads a problem out of the text as 3/8 +2/7 but then writes … and solves it as 3.8 + 2.7. She calls the commutative property the community property. She says proportion when she means ratio. She talks about denominators being equivalent when she means the fractions are equivalent.”
Hill, Kapitula, & Umland (2010, p. 820)
And she ranks in the second-to-top quartile on value-added.
![Page 44: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/44.jpg)
2013 William H. Angoff Memorial Lecture
MET Project Correlations with Classroom Observations
44
Subject Area
Classroom Observation
System
Correlation of Overall Quality
Rating with Prior-Year VAM Score
Mathematics CLASS 0.18 Mathematics FFT 0.13 Mathematics UTOP 0.27 Mathematics MQI 0.09 English Language Arts CLASS 0.08 English Language Arts FFT 0.07 English Language Arts PLATO 0.06
Bill & Melinda Gates Foundation (2012, pp. 46, 53)
![Page 45: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/45.jpg)
2013 William H. Angoff Memorial Lecture
MET Project Correlations with Classroom Climate
45
Same Year, Same Section
Same Year, Different Section
State Math Test .21 .22 State English Language Arts Test .10 .07
Balanced Assessment in Mathematics
.11 .11
Stanford 9 Open-Ended Reading .14 .06
![Page 46: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/46.jpg)
2013 William H. Angoff Memorial Lecture
Changes if a Different Achievement Test Is Used
46
Lockwood, et al. (2007) compared VAM scores using math “Procedures” versus “Problem Solving” subtests. Correlations ranged from .01 to .46, with median of .26.
“These correlations are uniformly low, … the two achievement outcomes lead to distinctly different estimates of teacher effects.”
Lockwood, et al. (2007, p. 54)
![Page 47: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/47.jpg)
2013 William H. Angoff Memorial Lecture
Changes if a Different Achievement Test Is Used
47
Papay (2007) compared VAM scores using three different reading tests, with similar results.
“Correlations between teacher value-added estimates derived from three separate reading tests … range from 0.15 to 0.58 …. if a school district were to reward teachers for their performance, it would identify a quite different set of teachers … depending simply on the specific reading assessment used.”
Papay (2011, p. 187)
![Page 48: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/48.jpg)
2013 William H. Angoff Memorial Lecture
MET Project Findings
48
Correlation between VAM scores based on two different math tests (same students, same year) = .38
Correlation between VAM scores based on two different English language arts tests (same students, same year) = .22
Bill & Melinda Gates Foundation (2012, pp. 23, 25)
![Page 49: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/49.jpg)
2013 William H. Angoff Memorial Lecture
State Tests vs. Content Standards
Too much memorization Too little complex reasoning Overall alignment indices low
49
Polikoff, Porter, & Smithson (2011)
Teaching to these tests will not foster desired range of student learning outcomes
![Page 50: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/50.jpg)
2013 William H. Angoff Memorial Lecture
Noncognitive Outcomes Work Attitudes and Behavior ◦ Punctuality ◦ Self-Discipline ◦ Listening ◦ Taking responsibility
Motivation, Goal Setting, Planning Problem Solving Teamwork, Positive Social Behavior Social and Emotional Skills Self-monitoring/Self-regulation
50
Examples from Levin (2012)
![Page 51: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/51.jpg)
2013 William H. Angoff Memorial Lecture
Nashville Teacher Survey From the POINT Experiment 80% - 85% of teachers agree that
“The POINT experiment ignores important aspects of my performance that are not measured by test scores.”
51
Springer, et al. (2010, p. 38)
![Page 52: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/52.jpg)
2013 William H. Angoff Memorial Lecture
Interpretive Argument
52
1. Scoring 3. Extrapolation
2. Generalization 4. Implication
This class, this year, this test
Bias?
Other classes, other years
Random Error?
Other tests, nontest outcomes Other Measures?
Validity of possible uses
Consequences?
![Page 53: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/53.jpg)
2013 William H. Angoff Memorial Lecture
How Should VAM Scores Be Used?
Inappropriate Uses ◦ Teacher VAM scores should not be included as a
substantial factor with a fixed weight in consequential teacher personnel decisions ◦ Teacher VAM scores should not be included as a
substantial factor in evaluations of principals ◦ Individual teachers’ VAM scores should not be
made public ◦ VAM scores should not be used to compare
teachers from very different sorts of schools, or working with very different student populations
53
![Page 54: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/54.jpg)
2013 William H. Angoff Memorial Lecture
Likely Consequences of Misuse Increased pressure to teach to the test Reduced cooperation among teachers
within a school Teacher resentment of students who
struggle with academic content Manipulation of assignments of students to
teachers ◦ Includes “push-out” of low achievers
54
![Page 55: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/55.jpg)
2013 William H. Angoff Memorial Lecture
How Should VAM Scores Be Used?
Appropriate Uses: For large-scale research studies ◦ Studying alternative teacher training programs,
educational policies, curricula, etc. Researchers should understand VAM well
55
![Page 56: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/56.jpg)
2013 William H. Angoff Memorial Lecture
How Should VAM Scores Be Used?
Appropriate Uses: For teacher evaluation, if: ◦ Scores based on sound, appropriate student
tests ◦ Comparisons limited to homogeneous teacher
groups ◦ No fixed weight—Flexibility to interpret VAM
scores in context for each individual case ◦ Users well trained to interpret scores ◦ Clear and accurate information about uncertainty
(e.g., “margin of error”)
56
![Page 57: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/57.jpg)
2013 William H. Angoff Memorial Lecture
Overview How big are teacher effects? A close look at test score gains How do VAMs work? Interpretive Argument ◦ What do scores mean? ◦ How reliable are they? ◦ What do they leave out? ◦ How should they be used?
Sound teacher evaluation
57
![Page 58: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/58.jpg)
2013 William H. Angoff Memorial Lecture
Sound Teacher Evaluation Attends to actual teaching practice Is research-based Provides constructive feedback to guide
improvement
58
![Page 59: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/59.jpg)
2013 William H. Angoff Memorial Lecture
Cautions re Classroom Observation
Requires observing multiple lessons Requires observer training May have poor reliability Susceptible to some of the same biases as
VAM
59
![Page 60: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/60.jpg)
2013 William H. Angoff Memorial Lecture
VAM as “Trigger”? Might work as stopgap, but limited… ◦ VAM scores available for only a minority
of teachers ◦ Weak teachers with high VAM scores will
be missed ◦ Makes teacher evaluation a remediation
issue
60
![Page 61: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/61.jpg)
2013 William H. Angoff Memorial Lecture
Sound Evaluation for All Teachers
61
All teachers have a right to expect sound professional evaluation and opportunities for continuous improvement, at all career stages The work of teaching children is far too important to settle for anything less.
![Page 62: William H. Angoff Memorial Lecture Inferences About Teachers Based on Student … · 2016. 5. 13. · 2013 William H. Angoff Memorial Lecture . The Allure of VAM Value-Added Models](https://reader034.vdocument.in/reader034/viewer/2022051920/600d3cb3f223ec4ad62c036b/html5/thumbnails/62.jpg)
62
Thank you