Inferences about School Quality using opportunity to learn data:

The effect of ignoring classrooms.

Felipe Martinez

CRESST/UCLA

CCSSO Large Scale Assessment Conference Boston, MA; June 21, 2004

Introduction 

We focus on two related issues concerning the valid use of measures of school performance in accountability systems: differences in achievement (and OTL) between classrooms, and the impact on measures of school quality of ignoring the classroom context.

Overview of research questions and studies.

1. Comparison of teacher and student reports of OTL2. Distribution and effects of OTL (as reported by

students and teachers) on student achievement in Reading.

3. Effect of ignoring classroom nesting in multilevel models. Effect on model-based measures of school quality.

Research Questions

• Comparison of teacher perceptions of the OTL they provide, to student perceptions of the OTL they receive.

• Factor Analysis

• Distribution and effects of OTL, as perceived by teachers and students.

• Multilevel Models (HLM)

• Distribution of student achievement and effect of ignoring classroom nesting on measures of school quality.

• Multilevel Models (HLM, Empirical Bayes estimates)

Factor AnalysisSample:

Our sample consisted of 97,675 students attending elementary schools in grades 2nd to 5th. In addition the sample included data from 6,902 teachers.

Methods:

A questionnaire was used to collect OTL data from students and their teachers in order to compare their perceptions of the educational activities that occur in the classroom during the school year.

The Student and Teacher OTL questionnaires were identical and consist of seven four-point Likert items related to content exposure and other classroom practices theoretically related to student performance in Language Arts. Each item inquires about the frequency with which the students performed a certain activity in the classroom during the school year. The scale ranges from 1 (almost never) to 4 (almost every day).

Factor AnalysisTeacher and Student reports of Opportunity to Learn

Score Distribution 1 2 3 4 S T S T S T S T 1. How often did you read literature? 2. How often did your teacher read aloud to you? 3. How often did you write compositions? 4. How often did you take notes on your ideas before beginning to write a composition? 5. How often did you use information from books to support ideas in your compositions? 6. How often did you rewrite your compositions to make them better? 7. How often did your teacher explain how your compositions would be scored?

3%

6%

3%

11%

9%

6%

8%

1%

1%

1%

1%

3%

1%

5%

3%

5%

11%

12%

13%

9%

13%

1%

3%

6%

7%

20%

13%

23%

13%

20%

37%

40%

32%

38%

31%

7%

21%

53%

63%

52%

66%

53%

80%

69%

49%

37%

46%

47%

48%

91%

75%

40%

29%

25%

20%

19%

Teachers Students

OTL Item Factor Loading Factor Loading

Student reads 0.244 Student Reading

1.000

Teacher reads

to students

Student

Reading 0.259 Teacher Activities

0.270

Student write 0.400 0.431

Student take notes 0.416 0.351

Student use info. 0.503 0.512

Student re-write

Student Writing

0.456

Student Writing

0.462

Teacher explain

grading criteria

Criteria 1.000 Teacher Activities

0.425

Factor AnalysisTeacher and Student reports of Opportunity to Learn

Factor AnalysisTeacher and Student reports of Opportunity to Learn

•Differences exist in the way teachers and their students perceive the nature of teaching and learning activities conducted in the classroom

•As Teachers, Students clearly separated activities related to writing; however, students regarded activities led by the teacher, independently of whether these involved reading aloud to them or explaining grading criteria, as part of a single construct of Teacher activities .

•This may reflect a certain degree of confusion in the students (or lack of direction from the teachers) in terms of the specific nature of the activity being carried out by the teacher—whereby students may for example perceive explanation of grading criteria as their teacher simply reading something to them.

Multilevel Analysis

Sample:

The sample for multilevel modeling includes 46,284 2nd to 5th grade students distributed across 4,972 classrooms (teachers), in 375 elementary schools

Methods: For each student achievement scores (SAT9 Reading) and background information was available, as well as context data about classrooms and schools. The factors created in the previous step are used as indicators of student and teacher OTL.

At this stage, three-level multilevel models (HLMs) were employed to correctly take into account the nested structure of the data—students nested within classrooms, which in turn were nested in schools.

Multilevel AnalysisUnconditional 2-Level Model

Level-1 (Student): ijjij

READNCE 0

, ij

~ N (0, 2)

Level-2 (School): j

uj 0000

, j

u0

~ N (0, )

Multilevel AnalysisUnconditional 3-Level Model

Level-1 (Student): ijkjkijk

READNCE 0

, ijk N (0, 2)

Level-2 (Classroom): jk

rkjk 0000

, jkr0 N (0, ) Level-3 (School):

ku

k 0000000 , ku00 N (0, )

Multilevel Analysis

Unconditional Two- and Three- Level Models of SAT9 Reading Scores

Random Effect Standard Variance Percent P-value

Deviation Component of Variance

2 Levels

Students, U0 8.359 69.884 79% 0.000

School, R 16.204 262.596 21%

3 Levels

Students, E 14.546 211.614 71%

Classroom, R0 7.119 50.692 18% 0.000

School, U00 5.803 33.678 11% 0.000

Random Effect Standard Variance Percent P-value

Deviation Component of Variance

2 Levels

Students, U0 8.359 69.884 79% 0.000

School, R 16.204 262.596 21%

3 Levels

Students, E 14.546 211.614 71%

Classroom, R0 7.119 50.692 18% 0.000

School, U00 5.803 33.678 11% 0.000

Multilevel Analysis

Unconditional Three- Level Models of Student OTL

Random Effect Standard Variance Percent P-value

Deviation Component of Variance

SOTL1 (Student Reading)

Students, E 0.58692 0.34448 80%

Classrooms, R0 0.28304 0.08011 18% 0.000

Schools, U00 0.05241 0.00275 2% 0.006

SOTL2 (Teacher Activities)

Students, E 1.01690 1.03409 60%

Classrooms, R0 0.78920 0.62283 37% 0.000

Schools, U00 0.17301 0.02993 3% 0.000

SOTL3 (Student Writing)

Students, E 1.80174 3.24627 59%

Classrooms, R0 1.44665 2.09279 38% 0.000

Schools, U00 0.34781 0.12097 3% 0.000

Fixed Effect Coefficient Standard

Error

T-ratio Approx.

d.f.

P-value

Model 1 Student variables alone

INTRCPT3, 000 46.2913 0.3590 128.944 336 0.000

Student Reading (SOTL1), 100 2.2170 0.1436 15.434 336 0.000

Teacher Activities (SOTL2),200 -0.2868 0.0856 -3.350 336 0.001

Student Writing (SOTL3), 300 0.1633 0.0465 3.506 336 0.001

Model 2 Teacher variables alone

For INTRCPT1, P0

INTRCPT3, 000 46.3313 0.3577 129.497 336 0.000

Student Reading (TOTL1), 010 0.6563 0.2517 2.608 336 0.009

Student Writing (TOTL1), 020 -0.0911 0.0989 -0.921 336 0.357

Criteria (TOTL1), 030 0.1588 0.2445 0.650 336 0.516

Multilevel AnalysisStudent- and Teacher- reported OTL as predictor of student

achievement

Multilevel AnalysisStudent- and Teacher- reported OTL as predictor of student

achievement

Fixed Effect Coefficient Standard

Error

T-ratio Approx.

d.f.

P-value

Model 3 Teacher and Student variables

For INTRCPT1, P0 46.3122 0.3573 129.611 336 0.000

Student OTL

Student Reading (SOTL1), 100 2.2056 0.1435 15.370 336 0.000

Teacher Activities (SOTL2),200 -0.2948 0.0863 -3.416 336 0.001

Student Writing (SOTL3), 300 0.1697 0.0470 3.611 336 0.001

Teacher OTL

Student Reading (TOTL1), 010 0.4498 0.2485 1.810 336 0.070

Student Writing (TOTL1), 020 -0.1704 0.0999 -1.706 336 0.088

Criteria (TOTL1), 030 0.1842 0.2442 0.754 336 0.451

Multilevel AnalysisVariation of Student-reported OTL effects across classrooms

Random Effect Standard

Deviation

Variance

Component

df Chi-

square

P-value

INTRCPT1, R0 6.945 48.234 559 2316.3 0.000

Student Reading (SOTL1), R1 1.070 1.146 559 890.51 0.000

Teacher Activities (SOTL2), R2 0.941 0.885 506 839.68 0.000

Student Writing (SOTL3), R3 0.664 0.441 559 889.69 0.000

Residual Variance, E 14.404 207.493

Multilevel AnalysisCorrelations of EB School residuals from unconditional

2- and 3-level models

2-Level (u0) 3-Level (u00) Reading Math Reading Math

Reading 1 2-Level (u0)

Math

.920

1

Reading

.991

.911

1

3-Level (u00)

Math

.909

.984

.919 1

Multilevel AnalysisConditional two-level model with school context

ijka

Pjka

jka

jkjkijkREADNCE Pijkijkijk ...

210 21

Level-1 (Student)

jkr

qjkX

jkX qkkkjk 0

...1 001000

Level-2 (School)

Multilevel AnalysisConditional three-level model with classroom

context

ijka

Pjka

jka

jkjkijkREADNCE Pijkijkijk ...

210 21

Level-1 (Student)

jkr

qjkX

jkX qkkkjk 0

...1 001000

Level-2 (Classroom)

00k = 000 + u00k

Level-2 (School)

Multilevel AnalysisCorrelations of EB School residuals from conditional

2- and 3-level models (considering classroom context)

2-Level (u0) 3-Level (u00) Reading Math Reading Math

Reading 1 2-Level (u0)

Math

.770

1

Reading

.277

.111

1

3-Level (u00)

Math

.215

.623

.330 1

 

Discussion

•Teachers and students do not necessarily perceive the same opportunities to learn within a classroom.

•Results are in agreement with previous research (e.g. see Muthen et. al., 1995) suggesting that OTL information collected from teachers may not add significantly to what is known from information collected from students.

•Students’ own perceptions of OTL are more closely linked to their achievement than are the perceptions of teachers of the opportunities they provide the same students.

•Although OTL is provided to student at the classroom level, measuring student perceptions may be a more powerful (and accurate) indicator of OTL than teacher reports.

 

Discussion

•Results also support the notion that the classroom environment is at least as important or more important than the larger school as determinant of student learning (see Kyriakydes, Campbell & Gagatsis, 2000; Hill & Rowe, 1996; Anderson, 1987, among others).

•Student OTL slopes vary significantly across classrooms (Level-2), but not across schools (Level-3). This implies that the exact effects of OTL at any given classroom can differ considerably from the average.

•Model-based measures of school quality (in our case Empirical Bayes school-level residuals) are impacted by model choice. Estimates for particular schools can differ considerably depending on whether the classroom environment is included by using a three-level model, or not.

 

Discussion

•Results emphasize the importance of using three-level models. In general, despite the fact that accountability systems are aimed at schools, results indicate that careful attention needs to be paid also to classroom differences within schools. Increasing attention to teacher effects is an encouraging sign although the use of these estimates in high stakes situations is problematic (McCaffrey et. al., 2004)

•Even within three-level models, however, use of EB residuals and other estimates of school quality for accountability purposes should be carefully considered as they constitute “at best, Type A effects” not suitable for accountability (Willms and Raudenbush, 1989; Raudenbush, 2004).

•Furthermore, the estimates of school performance we produce are cross-sectional in nature. For an up-to-date view on the promise but also the (sometimes overwhelming) complexity of the models needed for longitudinal studies of teacher and school effectiveness see the last issue of the Journal for Educational and Behavioral Statistics on Value Added Assessment (Spring 2004).