understanding value-added

35
Performance Management CPS Understanding Value- Added Lesson 2: How Value-Added Works

Upload: john-franklin

Post on 30-Dec-2015

44 views

Category:

Documents


1 download

DESCRIPTION

Understanding Value-Added. Lesson 2: How Value-Added Works. What is the Value-Added Metric?. Academic Growth = Student Learning. 2009. 2010. Value-Added is the District’s measure of elementary school growth. Value-Added is a nationally recognized way of measuring growth. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Understanding Value-Added

Performance Management CPS

Understanding Value-Added

Lesson 2: How Value-Added Works

Page 2: Understanding Value-Added

What is the Value-Added Metric?

Value-Added is the District’s measure of elementary school growth.

Value-Added is a nationally recognized way of measuring growth.

2

Emphasizes continual student improvement

Provides information to understand what drives continual improvement

20102009Academic Growth = Student Learning

Page 3: Understanding Value-Added

Measuring Growth, Not Attainment

3

200 210 220 230 240 250 260 270 280 290 300

(Year 2)

In this school, the percent meeting state standards is 25% in both Year 1 and Year 2.

Attainment is unchanged – but are students learning?

Analyzing growth provides this information

200 210 220 230 240 250 260 270 280 290 300

(Year 1)ISAT Scale Score

Meets S

tate Stan

dard

s

Page 4: Understanding Value-Added

Accounting for Student Populations

Student academic growth varies by grade, prior performance, and demographics.

The goal of the Value-Added metric is to measure the school’s impact on student learning independent of student demographic factors.

Value-Added accounts for the following student factors:

Controlling for the factors above gives proper credit for growth to low attainment schools and schools that serve unique populations.

4

Prior ISAT Reading Score Low-Income Status

Prior ISAT Math Score ELL Status

Grade Level IEP Status

Gender Homelessness

Race/Ethnicity Mobility

Page 5: Understanding Value-Added

How it Works

Value-Added is not a comparison to similar schools.

We do not look for a comparison group of schools that match each

other on all 9 student factors…such a group might not exist.

Rather, Value-Added compares growth of students in each school to

growth of students across the District, controlling for the list of student

factors.

To do this, we utilize a regression methodology, developed in

collaboration between CPS and academic experts from the University of

Wisconsin.

5

Page 6: Understanding Value-Added

All ISAT Math Scores for the District

Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.

6

125 175 225 275 3250

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

375

Page 7: Understanding Value-Added

3rd to 4th Grade ISAT Test Scores Only

Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.

7

125 175 225 275 3250

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

375

Page 8: Understanding Value-Added

This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.

It is downward-sloping because at higher levels of prior performance, average growth is smaller.

Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.

8

125 175 225 275 3250

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

375

Page 9: Understanding Value-Added

Repeat the process for each grade level.

Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.

9

125 175 225 275 3250

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

375

Page 10: Understanding Value-Added

Controlling for One Variable

10

Gain of 4th to 5th Grade students at a single school controlling for prior performance compared to the District average

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

This student grew faster than other 5th grade students with the same prior ISAT score.

This student grew slower.

Page 11: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

This line shows the average gain for all students from 4th to 5th grade.

Controlling for Multiple Variables

11

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 12: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

Controlling for Multiple Variables

Now we identify which students are English Language Learners

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 13: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

The blue line shows the average gain for ELL students between 4th and 5th grade.

Controlling for Multiple Variables

13

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 14: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

The orange line shows the average gain of non-ELL students between 4th and 5th grade.

Controlling for Multiple Variables

14

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 15: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

Controlling for Multiple Variables

15

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 16: Understanding Value-Added

Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

Controlling for Multiple Variables

16

Although this student grew slower than other 5th graders with the same pretest score, she grew faster than other ELL students with

the same pretest score.

150 175 200 225 2500

5

10

15

20

25

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 17: Understanding Value-Added

Now, we can control for other factors besides prior performance for Student A.

Controlling for Many Variables at Once

17

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 18: Understanding Value-Added

Based on Student A’s demographics, adjustments are made

Controlling for Many Variables at Once

18

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 19: Understanding Value-Added

Compared to similar students district-wide, Student A has above average gain

Controlling for Many Variables at Once

19

2009 ISAT Score

Sca

le S

core

Gai

n, 2

009

to 2

010

Page 20: Understanding Value-Added

Summary of Regression By measuring the impact of each student factor, the

regression model isolates the impact of the school isolates the impact of the school on student growth.

In other words, some growth is explained by external factors. We can measure the average impact of these external factors We can measure the average impact of these external factors on growth at the District level and subtract that impact from the school’s absolute growth.

The growth that is left over after removing the impact of these factors is attributed to the school. This is the value added value added by the school.

20

Page 21: Understanding Value-Added

Oak Tree Analogy

For an illustrative example of regression, view the “Oak Tree Analogy” presentation at:

http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/

The Oak Tree presentation illustrates the Value-Added model by using an analogy of two gardeners tending to oak trees.

21

Page 22: Understanding Value-Added

Some Things to Know Tested StudentsTested Students

All students making normal grade progression who took ISAT in both the previous year and current year are included in analysis.

Mobile StudentsMobile Students Mobile students count towards the Value-Added score in each school they

attended, but are weighted in the analysis by the amount of time they were in the school during the year.

English Language LearnersEnglish Language Learners ELL students in Program Years 0 through 5 are excluded from the analysis. This includes students who were in PY0-5 during the pretest year, even if they

have since exited the ELL program or moved to PY6.

Students with DisabilitiesStudents with Disabilities IEP status is differentiated by type of IEP. For example, the impact of a severe and profound disability is considered

separately from the impact of a speech and language disability.

22

Page 23: Understanding Value-Added

Value-Added Scores

Value-Added measures the difference between the growth of students at a school and the growth of similar students across the District.

23

Page 24: Understanding Value-Added

Standardization of Scores Growth on the ISAT is measured in ISAT scale score

points

However, one ISAT scale score point of growth is However, one ISAT scale score point of growth is more difficult to obtain in some grade levels than more difficult to obtain in some grade levels than others.others.

As a result, standardization is used to ensure that all Value-Added scores are on the same scale.

24

200 210 220 230 240

Student A “grew” by 35 ISAT scale score points

Page 25: Understanding Value-Added

Standardization of Scores Standardization is a common statistical process. In

this case, it is used to convert ISAT scale score points to a standard scale.

The unit of measure is the “standard deviation” which is a measure of distance from the mean.

i.e., how much does School A’s score deviate deviate from the mean?

This places all scores on the same scale, allowing for more precise comparisons between scores at different grade levels.

25

Page 26: Understanding Value-Added

The Standard Scale

26

Features of the Standard Scale The scale ranges from approximately -6 to 6. Zero (0) is the District average. About 68% of scores fall between -1 and 1. About 95% of scores fall between -2 and 2. About 99% of scores fall between -3 and 3. Only about 1% of scores are less than -3 or more than 3.

34% 34%

13.5% 2.5%2.5% 13.5%

Page 27: Understanding Value-Added

Reading the Value-Added Reports

27

Value-Added ScoreValue-Added ScorePercentile: This is the

percent of scores that fall below this score. Percentiles

range from 0th to 99th

Percentile: This is the percent of scores that fall

below this score. Percentiles range from 0th to 99th

Performance Category: This is based on the

percentile.

Performance Category: This is based on the

percentile.

Confidence Interval: This is explained in the next set of

slides.

Confidence Interval: This is explained in the next set of

slides.

Number of Students in the calculation: This is

weighted by the amount of time students were in the

school between the pretest and posttest.

Number of Students in the calculation: This is

weighted by the amount of time students were in the

school between the pretest and posttest.

Page 28: Understanding Value-Added

Confidence Intervals The Value-Added model controls for factors that CPS can

measure, but there are some factors that cannot be measured, such as:

Motivation to learn Family circumstances Health

In addition, the Value-Added model is a statistical estimation of the school’s impact on student learning and therefore contains a certain amount of random error.

For these reasons, the Value-Added model includes confidence intervalsconfidence intervals.

28

Page 29: Understanding Value-Added

Real World Example: Political Polling

29

A Political Polling company surveys a representative random sample of 1,000 community households about for whom they are going to vote on Election Day. The question they pose is:

If the election were held today, for whom would you cast your ballot?

The percentages of responses breakdown as follows: Candidate Jones would receive 54% of the vote Candidate Smith would receive 46% of the vote There is a +/- 3% margin of error

Page 30: Understanding Value-Added

Confidence Intervals in Political Polling

With the margin of error of +/- 3%, the range of the percentage of people who plan on

voting for each candidates is as follows:

Candidate Jones would receive between 51% and 57% of the vote.

43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57%

Candidate Smith would receive between 43% and 49% of the vote.

The confidence intervals do not overlap. Therefore the race is NOT “too close to call.” We can

predict with a high degree of confidence that Candidate Jones will win the race.

30

Page 31: Understanding Value-Added

A confidence interval is a range of scores around the Value-Added estimate.

We are 95% confident that the true Value-Added score falls within the confidence interval range.

The confidence interval is “n” dependent, meaning larger samples yield smaller confidence intervals.

This is because in larger samples, a score that is different from the average is less likely to be due to random error alone.

Confidence Intervals in Value-Added

31

1.01.0Example: 1.30.7

The Value-Added estimate is 1.0.

The confidence interval is ± 0.3.

The confidence interval range is from 0.7 to 1.3.

Page 32: Understanding Value-Added

Statistical Significance If the confidence interval does not include zerodoes not include zero, we say that the score is

statistically significantstatistically significant, meaning we are 95% confident that the score is different from zero.

A color is associated with each score based on the statistical significance:

Page 33: Understanding Value-Added

How Confidence Intervals are Reported

33

This is how Value-Added scores are displayed in the reports.

This school has a Value-Added score of -0.5 in reading

(the score is ½ of a standard deviation below the mean)

This school has a Value-Added score of -0.5 in reading

(the score is ½ of a standard deviation below the mean)

The confidence interval ranges from -1.9 to 0.8

Because the confidence interval includes zero, we say that this school is not statistically different from zero at the 95% confidence level.

For that reason, the bubble is yellow.

The confidence interval ranges from -1.9 to 0.8

Because the confidence interval includes zero, we say that this school is not statistically different from zero at the 95% confidence level.

For that reason, the bubble is yellow.

Page 34: Understanding Value-Added

Using Value-Added Information Performance Management

As an assessment of school performance To identify areas needing additional support or professional

development To identify best practice strategies for improving student growth

School Accountability (i.e., Performance Policy)

Additional Compensation Plans (i.e. Chicago TAP)

34

In all of these applications, Value-Added is used as just one additional piece of information, along with other data.

Page 35: Understanding Value-Added

For More Information

35

More lessons and other resources for understanding Value-Added are available at:

http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/

Lesson 2 (Part 2): Oak Tree Analogy

Lesson 3: Technical Specifications of the Value-Added Regression Model