Jim Lloyd_2007

Educational Value Added Educational Value Added Assessment System Assessment System

(EVAAS)(EVAAS)Olmsted Falls City Schools Initial Olmsted Falls City Schools Initial

Presentation of 4Presentation of 4thth Grade Grade StudentsStudents

Suppose this box represents Suppose this box represents a group of students for a a group of students for a

given grade level and given grade level and subject area… subject area…

Low

High

High

Low

… … and we fast forward to and we fast forward to the end of the given school the end of the given school

yearyear

An Achievement Status ConsiderationAn Achievement Status Consideration

Considerations for Gauging Effectiveness:

1) External Standard (proficiency level)

2) Student’s Ending Point of Achievement

? (Yes/No)

… sets a proficiency level students are expected to meet.

A Value-Added ConsiderationA Value-Added Consideration

Considerations for Gauging Effectiveness:

1) Student’s Prior Achievement Level

2) Student’s Ending Point of Achievement

… sets one end of the measurement at the students’ entry point, not at an external standard.

How much gain?

Factors related to student learning - Factors related to student learning - District, School, and Teacher Influence District, School, and Teacher Influence

on Student Progresson Student Progress

Following inferences were shared at the Following inferences were shared at the Governors Education Symposium (2004)Governors Education Symposium (2004)

Based on 22 years of Value-Added Study, Dr. Based on 22 years of Value-Added Study, Dr. Sanders draws the following conclusions:Sanders draws the following conclusions:

Variation in student academic progress can be Variation in student academic progress can be attributed this way:attributed this way:

5% attributed to District quality5% attributed to District quality

30% attributed to School quality 30% attributed to School quality

65% attributed to Teachers quality65% attributed to Teachers quality

Difference in student achievement levels after Difference in student achievement levels after comparably matched 2comparably matched 2ndnd graders experience qualitatively graders experience qualitatively

different teacher sequences of effectiveness, TNdifferent teacher sequences of effectiveness, TN

L=Least Effective H=Highly Effective (Sanders & Rivers, 1996)

Difference in student achievement levels after Difference in student achievement levels after Experiencing qualitatively different teacher Experiencing qualitatively different teacher

sequences of effectiveness, Dallassequences of effectiveness, Dallas

Source: Heather Jordan, Robert Mendro, & Dash Weerasinghe, “Teacher Effects On Longitudinal Student Achievement” 1997.

76

27

15

90

Beginning 3rd Grade

Score (Percentile)

Ave

rag

e M

ath

Sco

re 3

yrs

lat

er i

n P

erc

en

tile

s

Dallas StudentsAssigned To 3Highly EffectiveTeachers In ARow

Dallas StudentsAssigned To 3IneffectiveTeachers In ARow

Difference in student achievement levels after Difference in student achievement levels after experiencing qualitatively different levels of experiencing qualitatively different levels of

teacher effectiveness, Bostonteacher effectiveness, Boston

-0.6%

14.6%

0.3%

5.6%

-5

0

5

10

15

Average Student Growth

OverOne Year

(percent)

Least Effective TeachersMost Effective Teachers

Source: Boston Public Schools, in “High School Restructuring,” March 9, 1998.(courtesy of The Education Trust)

READING

MATH

Rivers (1999)Rivers (1999) Can varying sequences of teacher Can varying sequences of teacher

quality vary student chances of passing quality vary student chances of passing a high stakes test? a high stakes test? o Rivers (1999) followed students from 4Rivers (1999) followed students from 4thth through 9 through 9thth grade grade

from two large urban districts. from two large urban districts.

o Students were grouped into quartiles on the basis of Students were grouped into quartiles on the basis of achievement on the 4th grade standardized test.achievement on the 4th grade standardized test.

o Math teachers were identified for grades 5-8 and were Math teachers were identified for grades 5-8 and were linked to effectiveness levels (Low, Avg, High), based on linked to effectiveness levels (Low, Avg, High), based on their success in facilitating value-added achievement with their success in facilitating value-added achievement with a a priorprior group of students. group of students.

o Do these quality distinctions apply to another cohort of Do these quality distinctions apply to another cohort of students and offer reliable information about how a students and offer reliable information about how a student will do on a high stakes in the future?student will do on a high stakes in the future?

Probability of Passing High School Exam based on Sequence

of Teacher Effectiveness Experienced

Jim Lloyd_2006

How is value added different How is value added different than what we’re currently than what we’re currently

doing?doing? Current system measures only Current system measures only achievementachievement (what percentage (what percentage of kids pass the test).of kids pass the test).

New system measures New system measures progressprogress towards the standard, so can more towards the standard, so can more accurately reflect the growth of ALL accurately reflect the growth of ALL students, especially those who may students, especially those who may be hard-pressed to be hard-pressed to everever reach the reach the passing rate. passing rate.

Jim Lloyd_2006

How will this help me as an How will this help me as an educator?educator?

Measures growth from Measures growth from where the student where the student enters the grade level.enters the grade level.

Measures progress at Measures progress at building/grade levels.building/grade levels.

Shows if Shows if programs/instruction programs/instruction are impacting student are impacting student growth.growth.

Aligns classroom Aligns classroom resources to help all.resources to help all.

Jim Lloyd_2006

EVAASEVAAS

How to view a summary How to view a summary reportreport

Jim Lloyd_2006

Concepts to RememberConcepts to Remember This isn’t about proving…it’s about This isn’t about proving…it’s about

improving.improving.

The power of the scores is in the sample The power of the scores is in the sample size. size. The scores become less reliable as the size of The scores become less reliable as the size of

the population decreases.the population decreases.

We will not be looking at the individual We will not be looking at the individual growth of individual students because it is growth of individual students because it is not reliable.not reliable.

Jim Lloyd_2006

Colors to RememberColors to Remember

A score of “0” isn’t bad because we A score of “0” isn’t bad because we have a 0 growth model.have a 0 growth model.

RedRed isn’t good isn’t good YellowYellow is no detectable difference is no detectable difference GreenGreen is good is good

Jim Lloyd_2006

Terms Terms Mean NCE Gain—theMean NCE Gain—the average gain that average gain that

the group achieved compared to where the group achieved compared to where they started. This is the growth they started. This is the growth measure for the entire group of kids measure for the entire group of kids who have at least 2 data points.who have at least 2 data points.

Standard Error—aStandard Error—a statistical calculation statistical calculation as to how confident we are that the as to how confident we are that the score didn’t occur due to error.score didn’t occur due to error. Impacted by sample size and dispersionImpacted by sample size and dispersion Ohio’s model uses 1 SEM for significanceOhio’s model uses 1 SEM for significance

Jim Lloyd_2006

Terms cont.Terms cont.

State Base Year—theState Base Year—the average amount of average amount of growth that happened by the grade in growth that happened by the grade in Ohio (expressed in NCE units)Ohio (expressed in NCE units)

2005 Mean—the2005 Mean—the average amount of average amount of achievement that the achievement that the studentsstudents came in came in with.with.

2006 Mean—the2006 Mean—the average amount of average amount of achievement that the achievement that the studentsstudents left with. left with.

The amount of growth is the difference The amount of growth is the difference between the 2005 and 2006.between the 2005 and 2006.

Types of ReportsTypes of Reports

CompositeComposite Diagnostic Subject Specific Diagnostic Subject Specific

ReadingReading MathMath

Performance Subject SpecificPerformance Subject Specific ReadingReading MathMath

Subgroup specific (reading/math)Subgroup specific (reading/math)Jim Lloyd_2006

Composite ReportsComposite Reports

Jim Lloyd_2006

Jim Lloyd_2006

Composite ReportComposite Report

Shows the total amount of growth by Shows the total amount of growth by combining all achievement tests combining all achievement tests administered.administered.

We’re going to look at our composite We’re going to look at our composite report now.report now.

Jim Lloyd_2006

Building Summary ReportBuilding Summary Report

There are summary reports for each There are summary reports for each building as well that are subject building as well that are subject specific.specific.

Here is reading.Here is reading.

Jim Lloyd_2006

Subject Specific ReportsSubject Specific Reports

Jim Lloyd_2006

Jim Lloyd_2006

Subject Specific ReportsSubject Specific Reports

Shows the amount of growth Shows the amount of growth obtained in a particular subject area.obtained in a particular subject area.

We’re going to look at a Summary We’re going to look at a Summary Reading and Math Report now.Reading and Math Report now.

Jim Lloyd_2006

What is this? Growth standard of 0 is what we would expect if the student made a year’s growth

Jim Lloyd_2006

EVAASEVAAS

How to view a Diagnostic How to view a Diagnostic ReportReport

Jim Lloyd_2006

Pos Gain

Neg Gain

This shows us the # of OFCS students in each quintile

Total # of students in the testing pool is divided into 5 groups called quintiles

A “0 gain model” means that if students made about a year’s worth of growth we would see a gain within +1 or -1 of the standard error

Jim Lloyd_2006

Note that without at least 5 in a group, no data is reported

These students all have plenty of room to grow…even those at the 97th NCE.

Jim Lloyd_2006

Example—Scattered spread w/a low progress spike

Jim Lloyd_2006

Jim Lloyd_2006

EVAAS Math ReportsEVAAS Math Reports

District Reports 2006District Reports 2006

Current 6Current 6thth Grade Students Grade Students

Jim Lloyd_2006

Building Summary ReportBuilding Summary Report

There are summary reports for each There are summary reports for each building as well that are subject building as well that are subject specific.specific.

Here is mathematics.Here is mathematics.

Jim Lloyd_2006

Math Diagnostic ReportsMath Diagnostic Reports

What is the gain pattern?

Remember to look @ GAIN & SEM

Implications of EVAASImplications of EVAASQuestionsQuestions

What sorts of organizational What sorts of organizational strategies will lead to strategies will lead to increasing district, building and increasing district, building and teacher effectiveness?teacher effectiveness?

What are the professional What are the professional development implications of development implications of instituting EVAAS in Olmsted instituting EVAAS in Olmsted Falls?Falls?

What is the short and long What is the short and long term plan to prep the term plan to prep the administrative team and administrative team and building staff for this additional building staff for this additional accountability system?accountability system?

ResponsesResponses Teachers and principals make Teachers and principals make

the difference. Build capacity!the difference. Build capacity! Make a concerted effort to get Make a concerted effort to get

ahead of the curve and use this ahead of the curve and use this and other data to our and other data to our advantage.advantage.

Foster and reinforce a data-Foster and reinforce a data-driven culture across the district.driven culture across the district.

Build upon and expand our Build upon and expand our staffs’ capacity to implement staffs’ capacity to implement and monitor a standards-based and monitor a standards-based educational program for all educational program for all students.students.

Jim Lloyd_2006

Questions?Questions?