What Makes For a Good Teacher and Who Can Tell?
Douglas N. Harris Tim R. Sass
Dept. of Ed. Policy Studies Dept. of Economics
Univ. of Wisconsin Florida State Univ.
IES Research Conference, June 2008
Background Growing Interest in Linking Teacher Performance and
Compensation “Value Added” scores based on performance of a teacher’s
students have limitations Only available for teachers in tested subject (reading and math) Subject to measurement error and inter-temporal instability
Movement to Grant Principals Greater Autonomy in Personnel Decisions Charter schools School reform in New York City
Background Still Know Relatively Little About What Teacher
Characteristics are Associated With Ability to Improve Student Performance Most recent studies using panel data on individual student
achievement find that teacher experience (in the early years) matters, but little else in the way of readily observable characteristics seems to affect student achievement
Knowing what characteristics determine teacher quality has implications for teacher preparation, licensure, hiring practices and professional development
Research Questions Do principals’ evaluations of teachers contain
information beyond readily-observable “objective” teacher characteristics?
What factors determine a principal’s overall evaluation of her teachers?
What specific teacher characteristics are associated with a teacher’s ability to raise student achievement (ie. “value added”)?
Can principals do as well (or better) at predicting future teacher value added than past performance measures?
Literature Review Early Literature
Murnane (1975) Conditioning on student characteristics and prior achievement, a
one-standard-deviation increase in principal’s rating is associated with a 0.125 standard deviation increase in student third-grade math scores
Peterson (1987, 2000) Finds weak correlation between principal ratings of teachers and
parent or student satisfaction Milanowski (2004), Kimball, et al. (2004), White (2004)
Comparisons of principals’ four-point teacher evaluation scales with single-year estimates of teacher value added
Generally find weakly positive correlation between principal evaluations and estimated teacher quality (correlation = 0.2 - 0.4)
Literature Review Recent Literature
Jacob and Lefgren (2005) Consider both principal’s overall rating and evaluations of specific
teacher characteristics Find principals good at identifying best and worse teachers, but
less able to distinguish teachers in the middle Principal evaluations better at predicting future student
achievement than teacher experience or educational attainment, though generally not as good as value-added estimates
Principal evaluations better at predicting parental requests for teachers than teacher value-added scores
Data Anonymous mid-sized school district in Florida
Schools serve a diverse set of student populations Free/Reduced-Price Lunch: 5-90% Percent Minority: 10-90%
Longitudinal student-level achievement data Grades 1-10 1999/00 through 2004/05 Math and reading scores on the Stanford Achievement Test
Data Principal Interviews
In-person interviews conducted in Summer 2006 Single-blind evaluation of up to 10 teachers (5 per subject)
Random sample of teachers at school who taught at least three classes in the tested grades and subjects during 1999/00-2004/05
9-Point Rating Scale Not effective (1-3): performance is substantially below minimum standards Adequate: (4-6): meets standards but can improve in several areas. Exceptional (7-9): teacher is among the best I have seen
Asked to evaluate teachers on: Overall performance Ability to test scores Specific characteristics
Sample Studentand Teacher Characteristics
Math Reading
Total Students 76,308 70,916
Total Teachers 943 960
Principals Interviewed 30 30
Teachers Rated by Principal 234 231
Sample Teacher Grade Levelsand Principal Ratings
Math ReadingTeachers
Taught Primarily Elementary School 0.718 0.706Taught Primarily Middle School 0.152 0.153Taught Primarily High School 0.131 0.140
Mean Principal Rating of TeacherOverall 7.103 7.108Ability to Raise Test Scores 7.232 7.164“Caring” 7.397 7.468“Enthusiastic” 7.269 7.385“Motivated” 7.436 7.494“Strong Teaching Skills” 7.560 7.597“Knows Subject” 7.868 7.896“Communication Skills” 7.594 7.710“Intelligence” 7.897 7.922“Positive Relationship with Parents” 7.511 7.596“Positive Relationship with Students” 7.670
7.730
Pairwise Correlation of Principal’s Ratings ofTeachers With Teacher Characteristic Factors (Math)
Overall Rating
Ability to Raise Test
Scores
Inter-personal
SkillsMotivation/ Enthusiasm
Works Well With
Others
Knowledge/ Teaching
Skills
Overall Rating 1
Ability to Raise Test Scores
0.745** 1
Interpersonal Skills
0.703** 0.550** 1
Motivation/ Enthusiasm
0.738** 0.596** 0.734** 1
Works Well With Others
0.762** 0.598** 0.756** 0.732** 1
Knowledge/ Teaching Skills
0.881** 0.752** 0.612** 0.682** 0.644** 1
** Significant at the 0.05 level
Pairwise Correlation of Principal’s Ratings ofPairwise Correlation of Principal’s Ratings ofTeachers With Teacher Characteristic Factors (Reading)Teachers With Teacher Characteristic Factors (Reading)
Overall Rating
Ability to Raise Test
Scores
Inter-personal
SkillsMotivation/ Enthusiasm
Works Well With
Others
Knowledge/ Teaching
Skills
Overall Rating 1
Ability to Raise Test
Scores0.736** 1
Interpersonal Skills
0.719** 0.626** 1
Motivation/ Enthusiasm
0.699** 0.569** 0.714** 1
Works Well With Others
0.725** 0.589** 0.762** 0.677** 1
Knowledge/ Teaching
Skills0.861** 0.697** 0.644** 0.682** 0.650** 1
** Significant at the 0.05 level
Factor Loadings of Normalized Principal Ratings (Math)Factor Loadings of Normalized Principal Ratings (Math)
Teacher Characteristic Rated by Principal
Interpersonal Skills
Motivation/ Enthusiasm
Works Well With Others
Knowledge/ Teaching Skills
Intelligent -0.0481 0.0839 0.0606 0.7067
Works Well With Grade Team/Dept.
-0.0046 -0.0887 0.9711 0.0399
Works Well With Me (Principal)
0.1743 0.0835 0.7415 -0.0814
Positive Relationship With Parents
0.7231 0.0781 0.0768 0.0742
Positive Relationship With Students
0.9408 0.0103 -0.0131 0.0636
Caring 0.5591 0.1372 0.2422 -0.0185
Enthusiastic 0.1086 0.9721 -0.0707 -0.0035
Motivated 0.0398 0.5224 0.2802 0.1624
Strong Teaching Skills 0.1512 0.0258 -0.0462 0.8471
Knows Subject -0.0088 -0.0551 -0.0036 0.9831
Communication Skills 0.1040 0.1705 0.2734 0.3191
Factor Loadings of Normalized Principal Ratings (Reading)Factor Loadings of Normalized Principal Ratings (Reading)
Teacher Characteristic Rated by Principal
Interpersonal Skills
Motivation/ Enthusiasm
Works Well With Others
Knowledge/ Teaching Skills
Intelligent 0.0030 0.0055 0.0332 0.7182
Works Well With Grade Team/Dept.
0.0243 -0.0598 0.8522 0.0838
Works Well With Me (Principal)
0.1390 0.0461 0.8329 -0.0597
Positive Relationship With Parents
0.7554 0.0528 0.0649 0.0757
Positive Relationship With Students
0.9171 0.0271 0.0193 0.0313
Caring 0.6046 0.0983 0.2541 -0.0367
Enthusiastic 0.0775 0.9879 -0.0464 -0.0191
Motivated 0.0247 0.5366 0.2017 0.2445
Strong Teaching Skills 0.2237 0.0174 -0.0802 0.8140
Knows Subject -0.0834 -0.0156 0.0385 0.9819
Communication Skills 0.1650 0.2167 0.1715 0.3290
Estimation Procedure Estimate Student Achievement Model
Includes student, teacher and school fixed effects Estimated teacher effects yield within-school measure of teacher
effectiveness
Regress Estimated Teacher Fixed Effects on “Objective” Teacher Characteristics and Normalized Principal Ratings Objective characteristics include teacher experience,
possession of an advanced degree and certification status Weighted least squares estimates with the square root of
the numbers of students per teacher as weights
Value-Added Model
itmkiijmtititA PβXβ 21
Ait = achievement level of student i in year t
Xit = time-varying student characteristics
P-ijmt = time-varying characteristics of peers in classroom j in school m
i = student fixed-effect
k = teacher fixed effect
m = school fixed effect
Value-Added Results Used for Estimation of Teacher EffectsValue-Added Results Used for Estimation of Teacher Effects (Grades 2 – 10, 1999/2000 – 2004/05)(Grades 2 – 10, 1999/2000 – 2004/05)
Math Reading ________________________________________________________
Number of Schools Attended -2.131** -0.977(2.02) (0.84)
Attended Different School in Prior Year 1.910*** 0.834(2.99) (1.15)
Class Size -0.111** -0.038(2.21) (0.68)
Proportion of Classroom Peers -7.788*** -2.481 Who are White (4.37) (1.30)________________________________________________________
Covariance of Achievement Gain and:Student FE 0.186 0.183Teacher FE 0.032 0.100Model (Including School Indicators) 0.191 0.183Error 0.591 0.534
No. of Observations 76,308 70,916
________________________________________________________Note: Model Includes Student, Teacher, School and Grade-By-Year fixed effects. t-ratios adjusted for clustering at the classroom level.
Weighted Least Squares Estimates of the Relationship Between Weighted Least Squares Estimates of the Relationship Between Principal Ratings and Estimated Teacher Effectiveness Principal Ratings and Estimated Teacher Effectiveness
Math Reading
[1] [2] [1] [2]
_________________________________________________________
Overall Rating 2.685*** 1.661*(2.82) (1.76)
Ability to Raise Test Scores 2.570*** 0.975(4.52) (1.48)
_________________________________________________________
R-squared 0.078 0.149 0.061 0.057
No. of Observations 234 207 231 201_________________________________________________________
Note: Model includes a set of six experience category indicators and indicators for possession of advanced degrees and full certification.
WLS Estimates of the Relationship Between Principal Ratings and WLS Estimates of the Relationship Between Principal Ratings and Estimated Teacher Effectiveness -- By Grade Level Estimated Teacher Effectiveness -- By Grade Level
Math Reading
[1] [2] [1] [2]
________________________________________________________
Overall Rating 3.328*** 2.483**Elementary (2.84) (2.15)
Overall Rating 1.424 0.029Middle/High (0.87) (0.02)
Ability to Raise Test Scores 4.071** 2.971** Elementary (3.31) (2.25)
Ability to Raise Test Scores 1.456 -0.554 Middle/High (0.88) (0.32)
_______________________________________________________
R-squared 0.082 0.126 0.067 0.071
No. of Observations 234 199 231 201_______________________________________________________Note: Model includes a set of six experience category indicators and indicators for possession of advanced degrees and full certification.
WLS Estimates of the Relationship Between Teacher Subjective Characteristics and a Principal’s Overall Rating of Teachers
Math Reading_______________________________________________________
Interpersonal Skill 0.091* 0.192*** (1.86) (3.26)
Knowledge/Teaching Skills 0.592*** 0.592***
(14.53) (11.49)
Motivation/Enthusiasm 0.069 -0.002 (1.45) (0.04)
Works Well With Others 0.237*** 0.193***
(4.83) (3.30)_______________________________________________________
R-squared 0.853 0.786
No. of Observations 207 206_______________________________________________________
WLS Estimates of the RelationshipWLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness
Math Reading_______________________________________________________
Interpersonal Skill -0.250 1.656 (0.15) (0.93)
Knowledge/Teaching Skills 1.579 3.044*
(1.10) (1.96) Motivation/Enthusiasm -1.030 -1.532
(0.61) (0.96) Works Well With Others 2.523 -1.745
(1.46) (0.99) _______________________________________________________
R-squared 0.078 0.056
No. of Observations 207 206_______________________________________________________
WLS Estimates of the RelationshipWLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness (Math)
[1] [2] [3] [4]
_______________________________________________________
Interpersonal Skill 1.866*(1.94)
Knowledge/Teaching Skills 2.295** (2.38)
Motivation/Enthusiasm 1.661* (1.77)
Works Well With Others 2.572*** (2.63)
_______________________________________________________
R-squared 0.058 0.066 0.055 0.072
No. of Observations 207 207 207 207_______________________________________________________
WLS Estimates of the RelationshipWLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness (Reading)(Reading)
[1] [2] [3] [4]_______________________________________________________
Interpersonal Skill 1.167(1.15)
Knowledge/Teaching Skills 1.881* (1.84)
Motivation/Enthusiasm 0.559 (0.56)
Works Well With Others 0.521 (0.50)
_______________________________________________________
R-squared 0.035 0.045 0.030 0.030No. of Observations 206 206 206 206_______________________________________________________
Math Reading
[1] [2] [1] [2]______________________________________________________________________
Overall Rating 2.107*** 1.446*** (3.86) (2.69)
Teacher Fixed Effect (from 99/00-03/04) 0.230*** 0.106 (3.85) (1.44)
______________________________________________________________________
R-squared 0.236 0.237 0.115 0.114
No. of Observations 5361 5361 4399 4399______________________________________________________________________
Note: All models include controls for individual student mobility, class size, peer characteristics, student fixed effects (from 1999/00-2003/04), school indicators and a constant term.
Estimates of the Determinants of Student Achievement GainsEstimates of the Determinants of Student Achievement Gains(Grades 2 – 10, 2004/05)(Grades 2 – 10, 2004/05)
Summary Principal ratings of teachers predict teacher value-added even
after controlling for teacher experience, educational attainment and certification status
Principal ratings most strongly influenced by knowledge/teaching skill of teachers and the teacher’s ability to work with others
A teacher’s ability to raise test scores is also most strongly influenced by knowledge/teaching skill and ability to work with others Relationships imprecise when all factors included simultaneously
Principal’s overall evaluation does as well as past value-added at predicting current teacher effectiveness
Implications
Possibly greater role for principals in evaluating teachers and in retention decisions “pay for performance” scheme which includes principal
ratings likely better than current pay scale based on teacher experience and education alone
“value added” scores could be part of the mix as well
Possibly greater emphasis on teaching skills/knowledge in teacher preparation and professional development