Missouri Assessment Program

ELA Grades 3-8

Mathematics Grades 3-8

Science Grades 5 and 8

Spring 2017 Answer Change Report

Preliminary Report

Data Recognition Corporation

August 10, 2017

2

2017 MAP Answer Change Report

Overview

With the high-stakes nature of large-scale statewide assessment programs, there can be situations in which student responses, and hence their scores, may not be a true representation of students’ own abilities. Various activities may take place, such as a student copying from another student’s paper, students receiving inappropriate assistance before or during testing, or students’ responses being altered after testing. To maintain the integrity of Missouri MAP and the validity of the results, it is important that any such instances be discovered.

This study examines changes made to student responses to multiple choice items on the Spring 2017 MAP ELA, Mathematics, and Science assessments that occurred during the test-taking session. Of primary concern are instances of incorrect student responses that were changed to correct responses. We refer to these answer changes as wrong-to-right (WTR) answer changes. Inordinate numbers of WTR answer changes in a specifically identifiable testing administration group may indicate inappropriate intervention on students’ answer documents by an educator. Other answer changes include wrong-to-wrong (WTW) and right-to-wrong (RTW) changes.

Note that this document is a preliminary report and it includes results from the Spring 2017 answer change analysis. The final report will also include a summary of schools flagged for answer changes in both Spring 2016 and Spring 2017 test administrations.

DRC emphasizes that the results from this study should be used in conjunction with other information, and not as a standalone tool, to investigate whether inappropriate interventions may have taken place. Inordinate WTR answer change rates, by themselves, may simply be coincidental and do not necessarily indicate inappropriate behavior.

Method

The basis for these analyses is to count instances where one answer choice was replaced by a different answer choice. All operational and field test multiple-choice items were included in the analyses. Answer changes were captured during the online test administration. Paper-and-pencil, Braille, and large-print forms, from which student responses were transcribed into the online system by the examiner, were not included in the data analysis. The analyses were conducted for each test administration group, where a test administration group is a school indicated as a combination of a district code and a school code within a given content area and grade level.

A basic data cleaning was conducted to remove duplicate records. Also, students who did not respond to any MC items were removed from the data.

The answer change analyses include a statistical test of the null hypothesis (H0) that the mean number of answer changes of a given type (WTR, WTW, or RTW) for a test administration group constitutes a random sample from the state distribution of the same type of answer changes. The hypothesis is tested against the right-sided alternative (H1), that the mean number is too high to be explained by random sampling. Test administration groups for which H0 is rejected are flagged. The central limit theorem in

3

statistics tells us that the sampling distribution of mean number of a given type of answer changes for class i (mi) is asymptotically normal with mean and standard deviation

)( imMean (1)

i

in

mSD

)( (2)

where ni denotes the size and mi denotes the mean number of answer changes for test administration group i, respectively. In addition, μ and σ denote the mean and the standard deviation of the distribution of the number of answer changes of the population of individual students in MAP 2017.

Test administration groups were flagged if their mi was larger thanin

4 .

This flagging criterion of 4 standard deviations above the state mean provides a statistically conservative test. The standard normal table shows that under random sampling the asymptotic probability of observing a sample mean more than four standard deviations above the population mean is around 0.0001, or one in ten thousand. Even with this conservative test, rejection of H0 tells us only that the observed mean number of answer changes in a test administration group is unlikely to be the result of random sampling, and nothing beyond that with any certainty.

The denominator in the formula for the state standard deviation (equation 2) indicates that the flagging criterion for each test administration group is adjusted for the number of test takers in a test administration group with valid test scores. For example, if the state mean and SD of WTR answer changes are 1.73 and 2.11, respectively, the flagging criterion for a test administration group size of 20 is adjusted to 3.62 (i.e.,

62.3473.12011.2 ). This adjustment ensures that the flagging criterion is equally

stringent for test administration groups with considerably different numbers of test takers.

Results This section of the report provides results for WTR, WTW, and RTW changes.

Wrong-to-Right Changes

Table 1 provides the total number and percentage of test administration groups flagged for WTR answer changes for each grade and content area. Of the 5,418 test administration groups (schools) for ELA, a total of 193 (3.56%) groups were flagged for WTR answer changes larger than the flagging criterion across all grade levels. The highest percentage of schools was flagged in Grades 3 (4.10%) and the lowest percentage of schools was flagged in Grade 8 (less than 3%).

For Mathematics, a total of 238 (4.40%) test administration groups out of total 5,408 test administration groups were flagged across all test levels. A higher percentage of schools

4

was flagged in Grades 3 and 4 (6.23% and 5.37%, respectively) compared to higher grades. In Grades 6 and 8 less than 3% of schools were flagged for WTR answer changes.

For Science, of the 1,742 test administration groups, a total of 48 (2.76%) groups were flagged across Grades 5 and 8. Just over 3% of schools were flagged in Grade 5 and less than 2% of schools were flagged in Grade 8.

Overall, the percentage of test administration groups flagged for WTR answer changes larger than the flagging criterion was small across all grades and content areas, and it was comparable to the percentage of test administration groups flagged for WTR answer changes in Spring 2016.

A state summary of WTR answer changes of individual students (i.e., not test administration groups) is provided in Table 2. The summary includes means and standard deviations as well as the minimum, maximum, and selected percentiles of WTR answer changes. The mean number of WTR answer changes across grades for ELA ranges from 2.29 for Grade 8 to 2.88 for Grade 4 or approximately two to three WTR answer changes per student on average (see the column headed State Mean of WTR). For Mathematics tests the mean number of WTR answer changes across grades ranges from 1.54 for Grade 7 to 2.39 for Grade 3, or approximately two WTR changes per student on average. For Science tests, the mean number of WTR answer changes was 1.24 for Grade 5 and 1.25 for Grade 8, or approximately one WTR change per student on average.

These results are also comparable with the Spring 2016 results for all content areas.

A graphical representation of the average WTR changes for ELA, Mathematics, and Science are presented in Figures 1, 2, and 3, respectively. Wrong-to-Wrong Changes

Table 3 provides the total number and percentage of test administration groups flagged for WTW answer changes for each grade and content area. As can be seen in Table 3, a larger percentage of schools in lower grade levels was flagged for WTW compared to the higher-grade levels for both, ELA and Mathematics. In ELA, 6.86% of schools were flagged in Grade 3 and 7.15% of schools were flagged in Grade 4, while less than 4% of schools were flagged for WTW changes in Grades 7 and 8. Similarly, for Mathematics, 6.86% and 6.62% of schools were flagged for WTW changes in Grades 3 and 4, respectively, while less than 4% of schools were flagged in Grade 7 and less than 3% of schools were flagged in Grade 8. For Science, 4.72% of schools were flagged in Grade 5 and 3.52% of schools were flagged in Grade 8 for WTW changes.

A state summary of WTW changes made by individual students is provided in Table 4. The summary includes means and standard deviations as well as the minimum, maximum, and selected percentiles of WTW answer changes. The mean number of WTW answer changes across grades for ELA ranges from 1.44 to 2.24, or approximately one to two WTW answer changes per student on average (see the column headed State Mean of WTW). For Mathematics tests the mean number of WTW answer changes across grades ranges from 1.82 to 2.34, or approximately two WTW changes per student on average. For Science tests, the mean number of WTW answer changes was 0.71 for Grade 5 and 0.78 for Grade 8, or less than one WRW change per student on average.

5

The overall patterns of the WTW changes in Spring 2017 test administration is similar to the patterns of the WTW changes in Spring 2016 for all three content areas.

A graphical representation of the average WTW changes for ELA, Mathematics, and Science are presented in Figures 4, 5, and 6, respectively.

Right-to-Wrong Changes

Table 5 provides the total number and percentage of test administration groups flagged for RTW answer changes for each grade and content area. The percentage of schools flagged for RTW changes ranged from 1.05% (Grade 6) to 2.32% (Grade 3) for ELA and from 0.79% (Grade 6) to 2.49% (Grade 3) for Mathematics. The largest percentage of schools flagged for RTW changes was in Grade 3 in both content areas. For Science, approximately 1% of schools was flagged for RTW changes in both grades.

A state summary of RTW changes made by individual students is provided in Table 6. The summary includes means and standard deviations as well as the minimum, maximum, and selected percentiles of RTW answer changes. The mean number of RTW answer changes across all grades for ELA was approximately one answer change per student (see the column headed State Mean of RTW). For Mathematics tests the mean number of RTW answer changes was also approximately one change per student. For Science tests, the mean number of RTW answer changes less than one RTW change per student.

Again, when the mean number of RTW changes in Spring 2107 test administration was compared to the Spring 2016 results, it was observed that students made, on average, about the same number of RTW changes in Spring 2017 compared to the previous year.

A graphical representation of the average RTW changes for ELA, Mathematics, and Science are presented in Figures 7, 8, and 9, respectively.

Final Remarks This study examines student response changes to multiple choice items on the Spring 2017 MAP ELA, Mathematics, and Science assessments. A higher percentage of schools flagged for WTR changes in lower grade levels compared to higher grade levels was investigated to check whether younger students tended to make more answer changes in general. It was found that the patterns of WTW changes for ELA and Mathematics were similar to the patterns of WTR changes, with higher percentage of schools flagged for WTW changes in lower grade levels compared to higher grade levels. This finding supports, to some degree, a hypothesis that students in lower grades may not necessarily make more WTR changes, but rather students in lower grades may make more changes to their responses in general.

We repeat for emphasis that the results from this study should only be used in conjunction with other information to investigate whether inappropriate interventions may have taken place, particularly if the number of students in a test administration group is very small. Inordinate WTR answer change rates, by themselves, do not necessarily indicate inappropriate behavior.

6

Table 1. Number and Percentage of Test Administration Groups (Schools) Flagged for Wrong-to-Right Changes

Grade No. of Test

Administration

Groups

No. of Test Administration Groups

Flagged

Percentage of Test Administration Groups

Flagged

English Language Arts

3 1,123 46 4.10% 4 1,119 41 3.66% 5 1,059 37 3.49% 6 764 24 3.14% 7 669 25 3.74% 8 684 20 2.92%

Mathematics

3 1,123 70 6.23% 4 1,118 60 5.37% 5 1,062 48 4.52% 6 763 18 2.36% 7 668 24 3.59% 8 674 18 2.67%

Science

5 1,060 35 3.30% 8 682 13 1.91%

7

Table 2. State Summary of Wrong-To-Right (WTR) Answer Changes

Content Grade

No. of

MC items per

form

No. of Students

Clean N

State Sum of WTR

State Mean

of

WTR

State SD of WTR

State Min of

WTR

State Max of

WTR

State P25 of

WTR

State P50 of

WTR

State P75 of

WTR

State P99 of

WTR

State P99.9

of

WTR

ELA 3 33-34 65,822 65,815 176,333 2.68 2.24 0 27 1 2 4 10 15 ELA 4 37-39 66,305 66,305 191,222 2.88 2.35 0 36 1 2 4 11 17 ELA 5 38-39 65,243 65,239 176,034 2.70 2.25 0 33 1 2 4 10 16 ELA 6 35-37 64,347 64,344 166,825 2.59 2.13 0 31 1 2 4 9 14 ELA 7 32-35 64,456 64,454 159,697 2.48 2.13 0 21 1 2 4 9 13 ELA 8 32-35 63,929 63,919 146,674 2.29 2.04 0 24 1 2 3 9 13 MA 3 37-40 65,621 65,619 157,121 2.39 2.14 0 29 1 2 3 10 15 MA 4 36-43 66,114 66,113 151,070 2.29 2.05 0 33 1 2 3 9 16 MA 5 30-32 65,172 65,171 112,289 1.72 1.61 0 27 1 1 2 7 12 MA 6 32-35 64,587 64,584 104,982 1.63 1.49 0 23 1 1 2 6 9 MA 7 36-37 63,867 63,867 98,564 1.54 1.44 0 22 0 1 2 6 9 MA 8 37-40 51,426 51,422 100,808 1.96 1.70 0 20 1 2 3 7 10 SC 5 18 65,064 65,040 80,771 1.24 1.39 0 17 0 1 2 6 11 SC 8 15-16 64,129 64,096 79,946 1.25 1.27 0 12 0 1 2 5 7

8

Table 3. Number and Percentage of Test Administration Groups (Schools) Flagged for Wrong-to-Wrong Changes

Grade No. of Test

Administration

Groups

No. of Test Administration Groups

Flagged

Percentage of Test Administration Groups

Flagged

English Language Arts

3 1,123 77 6.86% 4 1,119 80 7.15% 5 1,059 67 6.33% 6 764 43 5.63% 7 669 24 3.59% 8 684 25 3.65%

Mathematics

3 1,123 77 6.86% 4 1,118 74 6.62% 5 1,062 55 5.18% 6 763 45 5.90% 7 668 25 3.74% 8 674 17 2.52%

Science

5 1,060 50 4.72% 8 682 24 3.52%

9

Table 4. State Summary of Wrong-To-Wrong (WTW) Answer Changes

Content Grade

No. of

MC items per

form

No. of Students

Clean N

State Sum of WTW

State Mean of WTW

State SD of WTW

State Min of WTW

State Max of WTW

State P25 of WTW

State P50 of WTW

State P75 of WTW

State P99 of WTW

State P99.9 of WTW

ELA 3 33-34 65,822 65,815 147,363 2.24 2.12 0 22 1 2 3 9 14 ELA 4 37-39 66,305 66,305 130,204 1.96 2.01 0 19 0 1 3 9 13 ELA 5 38-39 65,243 65,239 121,881 1.87 1.96 0 22 0 1 3 9 14 ELA 6 35-37 64,347 64,344 112,639 1.75 1.77 0 18 0 1 3 8 12 ELA 7 32-35 64,456 64,454 109,749 1.70 1.75 0 17 0 1 3 8 11 ELA 8 32-35 63,929 63,919 92,298 1.44 1.59 0 16 0 1 2 7 10 MA 3 37-40 65,621 65,619 150,226 2.29 2.19 0 23 1 2 3 10 15 MA 4 36-43 66,114 66,113 154,509 2.34 2.26 0 28 1 2 3 10 16 MA 5 30-32 65,172 65,171 121,750 1.87 1.84 0 19 0 1 3 8 12 MA 6 32-35 64,587 64,584 117,374 1.82 1.71 0 18 1 1 3 7 11 MA 7 36-37 63,867 63,867 131,641 2.06 1.88 0 24 1 2 3 8 12 MA 8 37-40 51,426 51,422 115,009 2.24 1.91 0 21 1 2 3 8 12 SC 5 18 65,064 65,040 45,860 0.71 1.02 0 13 0 0 1 4 7 SC 8 15-16 64,129 64,096 49,685 0.78 1.02 0 13 0 0 1 4 6

10

Table 5. Number and Percentage of Test Administration Groups (Schools) Flagged for Right-to-Wrong Changes

Grade No. of Test

Administration

Groups

No. of Test Administration Groups

Flagged

Percentage of Test Administration Groups

Flagged

English Language Arts

3 1,123 26 2.32% 4 1,119 20 1.79% 5 1,059 15 1.42% 6 764 8 1.05% 7 669 10 1.49% 8 684 12 1.75%

Mathematics

3 1,123 28 2.49% 4 1,118 14 1.25% 5 1,062 12 1.13% 6 763 6 0.79% 7 668 9 1.35% 8 674 8 1.19%

Science

5 1,060 10 0.94% 8 682 7 1.03%

11

Table 6. State Summary of Right-To-Wrong (RTW) Answer Changes

Content Grade

No. of

MC items per

form

No. of Students

Clean N

State Sum of RTW

State Mean of

RTW

State SD of RTW

State Min of RTW

State Max of RTW

State P25 of RTW

State P50 of RTW

State P75 of RTW

State P99 of RTW

State P99.9 of

RTW

ELA 3 33-34 65,822 65,815 64,127 0.97 1.15 0 14 0 1 2 5 7 ELA 4 37-39 66,305 66,305 62,355 0.94 1.13 0 13 0 1 1 5 7 ELA 5 38-39 65,243 65,239 64,886 0.99 1.13 0 14 0 1 2 5 7 ELA 6 35-37 64,347 64,344 57,320 0.89 1.07 0 14 0 1 1 4 6 ELA 7 32-35 64,456 64,454 65,254 1.01 1.14 0 11 0 1 2 5 7 ELA 8 32-35 63,929 63,919 56,399 0.88 1.07 0 12 0 1 1 4 6 MA 3 37-40 65,621 65,619 54,432 0.83 1.06 0 12 0 1 1 4 7 MA 4 36-43 66,114 66,113 55,571 0.84 1.05 0 15 0 1 1 4 6 MA 5 30-32 65,172 65,171 52,071 0.80 1.00 0 11 0 1 1 4 6 MA 6 32-35 64,587 64,584 45,158 0.70 0.91 0 10 0 0 1 4 5 MA 7 36-37 63,867 63,867 56,684 0.89 1.05 0 10 0 1 1 4 6 MA 8 37-40 51,426 51,422 54,496 1.06 1.14 0 11 0 1 2 5 7 SC 5 18 65,064 65,040 28,598 0.44 0.71 0 9 0 0 1 3 4 SC 8 15-16 64,129 64,096 37,974 0.59 0.82 0 7 0 0 1 3 5

12

Figure 1. MAP English Language Arts Average Answer Change: Wrong-to-Right

13

Figure 2. MAP Mathematics Average Answer Change: Wrong-to-Right

14

Figure 3. MAP Science Average Answer Change: Wrong-to-Right

15

Figure 4. MAP English Language Arts Average Answer Change: Wrong-to-Wrong

16

Figure 5. MAP Mathematics Average Answer Change: Wrong-to-Wrong

17

Figure 6. MAP Science Average Answer Change: Wrong-to-Wrong

18

Figure 7. MAP English Language Arts Average Answer Change: Right-to-Wrong

19

Figure 8. MAP Mathematics Average Answer Change: Right-to-Wrong

20

Figure 9. MAP Science Average Answer Change: Right-to-Wrong