understanding student achievement: the value of administrative data

30
Understanding Student Achievement: The Value of Administrative Data Eric Hanushek Stanford University

Upload: guiliaine-connor

Post on 02-Jan-2016

14 views

Category:

Documents


0 download

DESCRIPTION

Understanding Student Achievement: The Value of Administrative Data. Eric Hanushek Stanford University. Big Issues in School Policy Debates. Relating analysis to policy interests Confidence in causation Generalizability. Analytical designs. Random assignment experiments - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Understanding Student Achievement: The Value of Administrative Data

Understanding Student Achievement: The Value of Administrative Data

Eric HanushekStanford University

Page 2: Understanding Student Achievement: The Value of Administrative Data

Big Issues in School Policy Debates

Relating analysis to policy interests

Confidence in causation

Generalizability

Page 3: Understanding Student Achievement: The Value of Administrative Data

Analytical designs

Random assignment experiments Natural experiments “Data solutions”

Trade-offs Credibility Expense Questions that can be addressed

Page 4: Understanding Student Achievement: The Value of Administrative Data

UTD Texas Schools Project

Multiple cohorts followed 1993-2002

Annual achievement in grades 3-8 (TAAS math and reading)

Each cohort > 200,000 students in over 3,000 schools

Augmented with district data

Page 5: Understanding Student Achievement: The Value of Administrative Data

Examples of Topics Teacher quality variations Charter schools

Not discussed School choice and mobility Special education Teacher mobility Racial composition Peer achievement

Page 6: Understanding Student Achievement: The Value of Administrative Data

Existing Evidence on Teacher Quality

Substantial variation in teacher quality

Observable characteristics of teachers explain little of the variation

Salary and other factors affect teacher transition probabilities

No evidence on transitions and teacher quality

Page 7: Understanding Student Achievement: The Value of Administrative Data

Questions Addressed

What is variation in teacher quality? Measurable characteristics?

Do urban schools lose their best teachers? Quality by transitions

Do districts hire the best teachers?

Page 8: Understanding Student Achievement: The Value of Administrative Data

Basic model

standardized gain

( , )

isg

j

G

f X S

j j

Page 9: Understanding Student Achievement: The Value of Administrative Data

Measurement Error and Calculation of Variance of Teacher Quality

Observe teachers in two years:

Correlation across years:

(1) (2),j j

12( ) var( ) / var( )E r

Page 10: Understanding Student Achievement: The Value of Administrative Data

Estimated Variance in Teacher QualityLonestar District

Within districtWithin school

and year

unadjusteddemographic

controlsunadjusted

demographic controls

Teacher-year variation

0.210 0.179 0.109 0.104

Adjacent year correlation

0.500 0.419 0.458 0.442

Teacher quality variance / (s.d.)

0.105(0.32)

0.075(0.27)

0.050(0.22)

0.047(0.22)

Page 11: Understanding Student Achievement: The Value of Administrative Data

Kernal Density Estimates of Teacher Quality Distribution: Standardized Average Gains by Teacher Move Status

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

-2 -1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

Relative teacher quality (s.d.)

Stays at Campus Campus Change District Change Out of Public Education

Page 12: Understanding Student Achievement: The Value of Administrative Data

Conclusions on Teacher Quality Very large differences among teachers

Differences within schools much larger than between schools

Conventional measures not good index of quality (master’s degree, certification test)

Observable characteristics First year of experience Teacher-student race match

Common assumptions about market for teachers not correct Best do not leave Districts with advantages do not use them

Page 13: Understanding Student Achievement: The Value of Administrative Data

Popularity of charter schools

3,000 charter schools

40 states plus DC since 1991

1 percent of total students

10 percent of size of private school market

7+ percent rate of closure

Page 14: Understanding Student Achievement: The Value of Administrative Data

Evaluation issues

Most analysis of entry and participation

No reliable information on performance

Difficulty of selection issue

Very political

Page 15: Understanding Student Achievement: The Value of Administrative Data

Evaluation approaches

Model selection process [Heckman (1979)]

Instrument for attendance [Neal(1997)]

Intake randomization [Howell and Peterson (2002)]

Page 16: Understanding Student Achievement: The Value of Administrative Data

Difficulties with traditional approaches

Difficult to find factors affecting attendance but not achievement

Cannot handle treatment heterogeneity

Page 17: Understanding Student Achievement: The Value of Administrative Data

Empirical framework Mean differences in individual value-added

Identify charter school from individual entry-exit Consider time varying effects associated with

charter school movements

Heterogeneity across schools

Consumer responsiveness to quality

Page 18: Understanding Student Achievement: The Value of Administrative Data

Charter enrollment

1997 2001

4th grade 0.2 % 0.8%

7th grade 0.2% 0.9%

Page 19: Understanding Student Achievement: The Value of Administrative Data

Participation rates by race/ethnicity

1997 2001

Blacks 0.8% 2.2%

Hispanics 0.1% 0.6%

Whites 0.0% 0.4%

Low income 0.3% 0.8%

Page 20: Understanding Student Achievement: The Value of Administrative Data

Charters by vintage (analytical)

1997 1998 1999 2000 2001 2002 Total

one 17 10 70 83 43 47 270

Page 21: Understanding Student Achievement: The Value of Administrative Data

Charters by vintage (analytical)

1997 1998 1999 2000 2001 2002 Total

one 17 10 70 83 43 47 270two 2 16 9 69 78 40 214

Page 22: Understanding Student Achievement: The Value of Administrative Data

Charters by vintage (analytical)

1997 1998 1999 2000 2001 2002 Total

one 17 10 70 83 43 47 270two 2 16 9 69 78 40 214Three 0 2 15 8 68 73 166Four 0 1 2 15 8 66 92Five+ 0 0 1 3 17 22 43

Page 23: Understanding Student Achievement: The Value of Administrative Data

Charter school effect

Charter -0.17

Age 1 -0.33

Age 2 -0.25

Age 3 -0.08

Age 4 0.00

Age 5 or more 0.02

Page 24: Understanding Student Achievement: The Value of Administrative Data

Demographically Adjusted School Quality

Residual-Based Quality Measure

Charter Non-Charter

-1.57744 .788794

0

2.32064

Page 25: Understanding Student Achievement: The Value of Administrative Data

Do parents make good decisions?

Parents cannot see value added Considerable mobility/exiting

Models: Exit=f(quality, age, year, race, grade)

Page 26: Understanding Student Achievement: The Value of Administrative Data

Parental Choice(linear probability of exit)

Student characteristics

Student + peer

characteristics

Student + peer

characteristics + peer

achievement

School quality

0.002 0.006 0.006

School quality x charter

-0.152 -0.142 -0.138

Page 27: Understanding Student Achievement: The Value of Administrative Data

Parental Choice(linear probability of exit)

Student characteristics

Student + peer

characteristics

Student + peer

characteristics + peer

achievement

Student + peer

characteristics + peer

achievement

School quality

0.002 0.006 0.006

School quality x charter

-0.152 -0.142 -0.138

high income-0.187

low income-0.096

Page 28: Understanding Student Achievement: The Value of Administrative Data

Conclusions on Charter Schools

Difficult start-up period Mean performance regular ≈ charter

after two years Heterogeneity in both markets Parents react to quality in charter

market Low income reaction one half upper

income

Page 29: Understanding Student Achievement: The Value of Administrative Data

Administrative data

Pros Broader generalizability Understanding heterogeneity Perhaps less costly

Cons Requires structure (e.g., linearity, time pattern of

achievement) Regulatory problems (confidentiality) Data quality issues

Page 30: Understanding Student Achievement: The Value of Administrative Data

Papers on Teacher Quality and Charter Schools

www.hanushek.net or www.nber.org

Hanushek, Eric A., John F. Kain, Daniel M. O'Brien, and Steve G. Rivkin. 2005. "The market for teacher quality." National Bureau of Economic Research, Working Paper No. 11154, (February).

Hanushek, Eric A., John F. Kain, Steve G. Rivkin, and Gregory F. Branch. 2005. "Charter school quality and parental decision making with school choice." National Bureau of Economic Research, (March).