![Page 1: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/1.jpg)
Setting Performance Standards
Grades 5-7 NJ ASK
NJDOE
Riverside Publishing
May 17, 2006
![Page 2: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/2.jpg)
2
What do we mean by “Standard Setting”?• Content standards: specify the content that
should be learned by students – given implementation of the state standards and an opportunity to learn.
• Performance standards: specify the precise level of performance required on an assessment to achieve some category or outcome. How much must students know and be able to do?
![Page 3: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/3.jpg)
3
Proficiency Levels on NJ ASK
AdvancedProficient
PartiallyProficient Proficient
ProficientCut Score
AdvancedCut Score
100 - 199200 - 249
250 - 300
![Page 4: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/4.jpg)
4
NJ ASK Grades 5-7 Standard Setting
Phase 1 20 panelists on the Math Grades 5-7 & LAL
Grade 7 panels 19 panelists on the LAL Grades 5-6 panels Used a Modified Angoff procedure to
calculate cut scores in a 2-day workshop Phase 2
Representatives from Phase 1 and state and local policy makers reviewed results
Examined consistency across grades and subjects
![Page 5: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/5.jpg)
5
The Angoff Procedure• Research-based procedure used since
the early 1970s• The most commonly used standard
setting method — used in many other state testing programs as well as on certification tests
• Has undergone many modifications over the years and is often referred to as the Modified Angoff or Extended Angoff procedure
![Page 6: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/6.jpg)
6
Angoff Procedure (con’t.)• Original Angoff procedure asked
panelists to examine each item on a test and determine whether a student who was just barely Proficient would answer it correctly
• A common modification is to ask “If we had 100 barely Proficient students in the room, how many of them would answer this item correctly?”
![Page 7: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/7.jpg)
7
The Angoff Task
Review each item For MC items, determine how many out
of 100 just barely Proficient and just barely Advanced Proficient students should answer each item correctly
For OE items, determine the average score a just barely student at each performance level should receive
![Page 8: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/8.jpg)
8
Validity Considerations The process was well-planned – see
standard setting proposal. All policy, operational and technical
decisions were reasonable and well-documented.
The judges were satisfied with the process and its results – see the evaluation results.
![Page 9: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/9.jpg)
9
The Final Decision on Cut Scores – Reminder
Legitimate authority should have all the information they want or need to make an informed decision
At a minimum it should include: Overall results – Phase 1 and Phase 2 Variance in judgments and in measurement Impact on subpopulations in the state Impact on accountability Documentation of the process (with as
much detail as is necessary)
![Page 10: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/10.jpg)
10
Ratings Cut scores are determined over three
rounds of ratings Ratings were made independently After each set of ratings, panelists
saw their cut scores and the average cut scores
There was a discussion between each round
![Page 11: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/11.jpg)
11
Calculating the Cut Score Consider the MC ratings as probabilities We sum the probabilities to obtain a cut score for
the MC items and add the average scores for the OE items to get the total cut score
Item RatingPossible1 .85 12 .55 13 .40 14 1.5 3
Total= .85 + .55 + .40 + 1.5 = 3.3 points out of a possible 6 points
![Page 12: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/12.jpg)
12
“Just Barely” Performance The idea of just barely performance
is key to the Angoff procedure We spent time talking about what it
means to be just barely Proficient and Advanced Proficient
Our discussions were driven by the NJ ASK LAL and Math Grades 5-7 Performance Level Descriptors (PLDs)
![Page 13: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/13.jpg)
13
Phase 2 Meeting Policymakers and stakeholders gathered
to review the results of Phase 1 in light of NJ ASK program goals.
Reviewed results from Phase 1 along with standard deviation of judgments and impact data.
Compared results of Grades 3, 4 & 8 LAL and Math with Grades 5-7 LAL and Math and discussed the need for consistency across grades and content areas.
![Page 14: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/14.jpg)
14
Phase 2 Results: Final Cut Score Recommendations Committee produced cut score recommendations for
NJDOE and the State Board:
ProficientAdvanced Proficient Proficient
Advanced Proficient
Grade 5 16 30 18 30Grade 6 20 32 17 31Grade 7 21 34 13 26
LAL Math
![Page 15: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/15.jpg)
15
Evidence the Results Were Technically Sound
Survey Results (see handout) Standard deviation of judgments in each round:
ProficientAdvanced Proficient Proficient
Advanced Proficient
Round1 2.6 2.1 3.2 3.12 2.2 2.4 2.6 2.53 1.6 1.7 0.4 0.4
1 2.5 3.3 2.0 2.42 2.9 4.4 1.6 0.23 0.6 1.4 0.7 0.9
1 2.2 2.0 2.6 2.62 3.5 3.4 2.0 2.33 2.4 2.2 0.8 1.1
SD of Judgments
Grade 5
Grade 6
Grade 7
LAL Math
SD of Judgments
![Page 16: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/16.jpg)
16
Distribution of Students by SubgroupLAL Grade 5
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 8.1 6.7 9.6 11.3 1.8 2.0 19.0 10.0 1.0 0.6 2.0
P 70.7 70.3 71.3 77.4 59.6 62.4 73.0 76.5 50.2 42.0 60.2
PP 21.2 23.0 19.1 11.3 38.6 35.6 8.0 13.5 48.8 57.4 37.8
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 17: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/17.jpg)
17
Distribution of Students by SubgroupLAL Grade 6
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 7.4 5.8 9.1 10.7 1.0 1.7 15.0 8.9 0.9 1.0 1.3
P 56.5 55.1 58.2 66.2 41.2 42.1 62.4 63.0 32.1 22.2 40.9
PP 36.1 39.1 32.7 23.1 57.8 56.2 22.6 28.1 67.0 76.8 57.8
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 18: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/18.jpg)
18
Distribution of Students by SubgroupLAL Grade 7
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 5.7 3.9 7.5 8.1 1.5 1.2 12.2 6.9 0.6 0.5 1.2
P 56.8 54.3 59.5 67.3 39.9 41.8 69.2 63.9 31.0 18.8 38.4
PP 37.5 41.8 33.0 24.6 58.6 57.0 18.6 29.2 68.4 80.7 60.4
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 19: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/19.jpg)
19
Distribution of Students by SubgroupMath Grade 5
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 22.6 24.0 21.2 29.4 7.7 10.2 48.9 26.7 7.0 7.6 9.5
P 56.0 53.5 58.7 59.1 51.6 54.8 45.1 59.2 44.4 40.9 51.4
PP 21.4 22.5 20.1 11.5 40.7 35.0 6.0 14.1 48.6 51.5 39.1
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 20: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/20.jpg)
20
Distribution of Students by SubgroupMath Grade 6
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 13.4 14.4 12.6 18.1 3.3 4.3 32.2 16.1 2.5 3.0 3.8
P 53.0 51.6 54.6 62.5 36.6 41.5 54.4 59.0 28.7 25.5 38.0
PP 33.6 34.0 32.8 19.4 60.1 54.2 13.4 24.9 68.8 71.5 58.2
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 21: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/21.jpg)
21
Distribution of Students by SubgroupMath Grade 7
0%
20%
40%
60%
80%
100%
AP
P
PP
AP 11.8 13.5 10.2 16.7 2.2 2.7 30.6 14.4 1.4 2.1 2.5
P 47.6 46.5 48.9 56.9 31.1 37.2 52.2 53.7 22.0 24.7 32.5
PP 40.6 40.0 40.9 26.4 66.7 60.1 17.2 31.9 76.6 73.2 65.0
Total Male Female White Black H A Gen ED SP ED LEP EC Dis
![Page 22: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/22.jpg)
22
LALPhase 2 Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Per
cen
t in
Lev
el
AP
P
PP
AP 4.4 4.4 8.1 7.4 5.7 8.0
P 78.8 77.2 70.7 56.5 56.8 64.3
PP 16.7 18.4 21.2 36.1 37.5 27.7
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Final Recommended CutscoresLAL Grades 5-7 (with LAL Grades 3,4 &8)
![Page 23: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/23.jpg)
23
MathPhase 2 Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Per
cen
t in
Lev
el
AP
P
PP
AP 28.8 31.7 22.6 13.4 11.8 18.7
P 53.7 48.5 56.0 53.0 47.6 43.7
PP 17.5 19.8 21.4 33.6 40.6 37.6
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Final Recommended CutscoresMath Grades 5-7 (with Math Grades 3,4 &8)
![Page 24: Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d085503460f949da10d/html5/thumbnails/24.jpg)
24
Key Statistics from Surveys