understanding value-added
DESCRIPTION
Understanding Value-Added. Lesson 2: How Value-Added Works. What is the Value-Added Metric?. Academic Growth = Student Learning. 2009. 2010. Value-Added is the District’s measure of elementary school growth. Value-Added is a nationally recognized way of measuring growth. - PowerPoint PPT PresentationTRANSCRIPT
Performance Management CPS
Understanding Value-Added
Lesson 2: How Value-Added Works
What is the Value-Added Metric?
Value-Added is the District’s measure of elementary school growth.
Value-Added is a nationally recognized way of measuring growth.
2
Emphasizes continual student improvement
Provides information to understand what drives continual improvement
20102009Academic Growth = Student Learning
Measuring Growth, Not Attainment
3
200 210 220 230 240 250 260 270 280 290 300
(Year 2)
In this school, the percent meeting state standards is 25% in both Year 1 and Year 2.
Attainment is unchanged – but are students learning?
Analyzing growth provides this information
200 210 220 230 240 250 260 270 280 290 300
(Year 1)ISAT Scale Score
Meets S
tate Stan
dard
s
Accounting for Student Populations
Student academic growth varies by grade, prior performance, and demographics.
The goal of the Value-Added metric is to measure the school’s impact on student learning independent of student demographic factors.
Value-Added accounts for the following student factors:
Controlling for the factors above gives proper credit for growth to low attainment schools and schools that serve unique populations.
4
Prior ISAT Reading Score Low-Income Status
Prior ISAT Math Score ELL Status
Grade Level IEP Status
Gender Homelessness
Race/Ethnicity Mobility
How it Works
Value-Added is not a comparison to similar schools.
We do not look for a comparison group of schools that match each
other on all 9 student factors…such a group might not exist.
Rather, Value-Added compares growth of students in each school to
growth of students across the District, controlling for the list of student
factors.
To do this, we utilize a regression methodology, developed in
collaboration between CPS and academic experts from the University of
Wisconsin.
5
All ISAT Math Scores for the District
Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.
6
125 175 225 275 3250
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
375
3rd to 4th Grade ISAT Test Scores Only
Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.
7
125 175 225 275 3250
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
375
This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.
It is downward-sloping because at higher levels of prior performance, average growth is smaller.
Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.
8
125 175 225 275 3250
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
375
Repeat the process for each grade level.
Regression LinesRegression shows how growth relates to another variable—in this case prior performance on the ISAT.
9
125 175 225 275 3250
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
375
Controlling for One Variable
10
Gain of 4th to 5th Grade students at a single school controlling for prior performance compared to the District average
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
This student grew faster than other 5th grade students with the same prior ISAT score.
This student grew slower.
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
This line shows the average gain for all students from 4th to 5th grade.
Controlling for Multiple Variables
11
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
Controlling for Multiple Variables
Now we identify which students are English Language Learners
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
The blue line shows the average gain for ELL students between 4th and 5th grade.
Controlling for Multiple Variables
13
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
The orange line shows the average gain of non-ELL students between 4th and 5th grade.
Controlling for Multiple Variables
14
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
Controlling for Multiple Variables
15
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.
Controlling for Multiple Variables
16
Although this student grew slower than other 5th graders with the same pretest score, she grew faster than other ELL students with
the same pretest score.
150 175 200 225 2500
5
10
15
20
25
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Now, we can control for other factors besides prior performance for Student A.
Controlling for Many Variables at Once
17
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Based on Student A’s demographics, adjustments are made
Controlling for Many Variables at Once
18
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Compared to similar students district-wide, Student A has above average gain
Controlling for Many Variables at Once
19
2009 ISAT Score
Sca
le S
core
Gai
n, 2
009
to 2
010
Summary of Regression By measuring the impact of each student factor, the
regression model isolates the impact of the school isolates the impact of the school on student growth.
In other words, some growth is explained by external factors. We can measure the average impact of these external factors We can measure the average impact of these external factors on growth at the District level and subtract that impact from the school’s absolute growth.
The growth that is left over after removing the impact of these factors is attributed to the school. This is the value added value added by the school.
20
Oak Tree Analogy
For an illustrative example of regression, view the “Oak Tree Analogy” presentation at:
http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/
The Oak Tree presentation illustrates the Value-Added model by using an analogy of two gardeners tending to oak trees.
21
Some Things to Know Tested StudentsTested Students
All students making normal grade progression who took ISAT in both the previous year and current year are included in analysis.
Mobile StudentsMobile Students Mobile students count towards the Value-Added score in each school they
attended, but are weighted in the analysis by the amount of time they were in the school during the year.
English Language LearnersEnglish Language Learners ELL students in Program Years 0 through 5 are excluded from the analysis. This includes students who were in PY0-5 during the pretest year, even if they
have since exited the ELL program or moved to PY6.
Students with DisabilitiesStudents with Disabilities IEP status is differentiated by type of IEP. For example, the impact of a severe and profound disability is considered
separately from the impact of a speech and language disability.
22
Value-Added Scores
Value-Added measures the difference between the growth of students at a school and the growth of similar students across the District.
23
Standardization of Scores Growth on the ISAT is measured in ISAT scale score
points
However, one ISAT scale score point of growth is However, one ISAT scale score point of growth is more difficult to obtain in some grade levels than more difficult to obtain in some grade levels than others.others.
As a result, standardization is used to ensure that all Value-Added scores are on the same scale.
24
200 210 220 230 240
Student A “grew” by 35 ISAT scale score points
Standardization of Scores Standardization is a common statistical process. In
this case, it is used to convert ISAT scale score points to a standard scale.
The unit of measure is the “standard deviation” which is a measure of distance from the mean.
i.e., how much does School A’s score deviate deviate from the mean?
This places all scores on the same scale, allowing for more precise comparisons between scores at different grade levels.
25
The Standard Scale
26
Features of the Standard Scale The scale ranges from approximately -6 to 6. Zero (0) is the District average. About 68% of scores fall between -1 and 1. About 95% of scores fall between -2 and 2. About 99% of scores fall between -3 and 3. Only about 1% of scores are less than -3 or more than 3.
34% 34%
13.5% 2.5%2.5% 13.5%
Reading the Value-Added Reports
27
Value-Added ScoreValue-Added ScorePercentile: This is the
percent of scores that fall below this score. Percentiles
range from 0th to 99th
Percentile: This is the percent of scores that fall
below this score. Percentiles range from 0th to 99th
Performance Category: This is based on the
percentile.
Performance Category: This is based on the
percentile.
Confidence Interval: This is explained in the next set of
slides.
Confidence Interval: This is explained in the next set of
slides.
Number of Students in the calculation: This is
weighted by the amount of time students were in the
school between the pretest and posttest.
Number of Students in the calculation: This is
weighted by the amount of time students were in the
school between the pretest and posttest.
Confidence Intervals The Value-Added model controls for factors that CPS can
measure, but there are some factors that cannot be measured, such as:
Motivation to learn Family circumstances Health
In addition, the Value-Added model is a statistical estimation of the school’s impact on student learning and therefore contains a certain amount of random error.
For these reasons, the Value-Added model includes confidence intervalsconfidence intervals.
28
Real World Example: Political Polling
29
A Political Polling company surveys a representative random sample of 1,000 community households about for whom they are going to vote on Election Day. The question they pose is:
If the election were held today, for whom would you cast your ballot?
The percentages of responses breakdown as follows: Candidate Jones would receive 54% of the vote Candidate Smith would receive 46% of the vote There is a +/- 3% margin of error
Confidence Intervals in Political Polling
With the margin of error of +/- 3%, the range of the percentage of people who plan on
voting for each candidates is as follows:
Candidate Jones would receive between 51% and 57% of the vote.
43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57%
Candidate Smith would receive between 43% and 49% of the vote.
The confidence intervals do not overlap. Therefore the race is NOT “too close to call.” We can
predict with a high degree of confidence that Candidate Jones will win the race.
30
A confidence interval is a range of scores around the Value-Added estimate.
We are 95% confident that the true Value-Added score falls within the confidence interval range.
The confidence interval is “n” dependent, meaning larger samples yield smaller confidence intervals.
This is because in larger samples, a score that is different from the average is less likely to be due to random error alone.
Confidence Intervals in Value-Added
31
1.01.0Example: 1.30.7
The Value-Added estimate is 1.0.
The confidence interval is ± 0.3.
The confidence interval range is from 0.7 to 1.3.
Statistical Significance If the confidence interval does not include zerodoes not include zero, we say that the score is
statistically significantstatistically significant, meaning we are 95% confident that the score is different from zero.
A color is associated with each score based on the statistical significance:
How Confidence Intervals are Reported
33
This is how Value-Added scores are displayed in the reports.
This school has a Value-Added score of -0.5 in reading
(the score is ½ of a standard deviation below the mean)
This school has a Value-Added score of -0.5 in reading
(the score is ½ of a standard deviation below the mean)
The confidence interval ranges from -1.9 to 0.8
Because the confidence interval includes zero, we say that this school is not statistically different from zero at the 95% confidence level.
For that reason, the bubble is yellow.
The confidence interval ranges from -1.9 to 0.8
Because the confidence interval includes zero, we say that this school is not statistically different from zero at the 95% confidence level.
For that reason, the bubble is yellow.
Using Value-Added Information Performance Management
As an assessment of school performance To identify areas needing additional support or professional
development To identify best practice strategies for improving student growth
School Accountability (i.e., Performance Policy)
Additional Compensation Plans (i.e. Chicago TAP)
34
In all of these applications, Value-Added is used as just one additional piece of information, along with other data.
For More Information
35
More lessons and other resources for understanding Value-Added are available at:
http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/
Lesson 2 (Part 2): Oak Tree Analogy
Lesson 3: Technical Specifications of the Value-Added Regression Model