1 designing high quality, affordable assessment systems edward roeber michigan state university...
TRANSCRIPT
1
Designing High Quality, Affordable Assessment Systems
Edward RoeberMichigan State University
National Research Council Board on Testing and Assessment
April 6-7, 2010
2
Overview
Balanced Assessment Systems Horizontally Balanced Vertically Balanced
Summative Assessment Designs Purpose of this Study Typical Current Assessment Program High Quality Assessment Program
Development and Administration Costs Cost Reduction Strategies Results Recommendations
3
Vertically-Balanced System
Summative Component Broad array of types of assessment
Interim Benchmark Component Instructionally-relevant, short-cycle summative assessments
also using a broad array of assessment types
Formative Assessment Component Adequate professional development High-quality pre-service education
All three components present, and equally important to the teacher and the student
4
Vertically-Balanced System
Portions of this system are out of balance: Interim benchmark assessments not thoughtfully created,
acquired, or used Educators don’t understand how to use interim assessments Only conventional types of assessment used Lack of understanding about what formative assessment is and
is not (e.g., it is not an item bank) Lack of opportunities for educators to learn how to use formative
assessment strategies as they teach Educators don’t learn about classroom and formative
assessment in pre-service programs State summative assessments overpower the interim and
formative assessments in the minds of educators (and students)
5
Horizontally-Balanced Systems
Multiple-choice items are the predominant form of assessment used by states
Some states use constructed-response items Fewer still use performance assessments Emphasis is on speed of return of results, not quality
or usefulness of the information once it gets there Schools feel compelled to “teach to the test” in some
not-so-good ways - some subjects not taught at all and the rest are taught to the extent that they are on the test
Only state-assessed skills are focused on in local assessments
6
Horizontally-Balanced Systems
This system is out of balance in some important ways More types of items should be used in the system
• Short- and extended-response items• Performance events• Performance tasks
This broader array of items could have a positive impact on teaching and learning - could be assessments worth teaching to
These state assessments could also serve as a model for developing instructionally-sensitive interim assessments
The prime question is can states can afford to use this broader array of assessment?
7
Purpose of This Cost Study
Determine the cost of a typical state assessment program that primarily uses multiple-choice items
Design a high quality summative assessment (HQA) as just described
Note - HQA designed for Mathematics and Reading/Writing only. This excludes Science, alternate assessments for students with disabilities and ELPA assessments
Determine the cost of such a HQA Determine whether several potential strategies could reduce costs
of the HQA significantly Present various options and costs for an interim assessment
system similar to the HQA
8
Current Typical Summative Assessment Design
Summative Assessment
Mathematics Multiple ChoiceShort Constructed
ResponseExtended Constructed
ResponsePerformance
Event Performance Task
Current Typical
Assessment 50 0 2 0 0
Summative Assessment
English Language Arts Multiple ChoiceShort Constructed
ResponseExtended Constructed
ResponsePerformance
Event Performance Task
Current Typical
Assessment (Reading) 50 0 2 0 0
Current Typical
Assessment (Writing)* 10 0 1 0 0
Summative Assessment Design
Item Counts
Item Counts
9
Current Typical Interim Assessment Design
Interim Assessment
Mathematics Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
Current Typical
Assessment 40 0 0 0 0
Interim Assessment
English Language Arts Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
Current Typical
Assessment 40 0 0 0 0
Item Counts
Item Counts
Interim Assessment Design
10
High Quality Summative Assessment Design
Summative Assessment
Mathematics Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
High Quality 2 2 2
Assessment 25 (1 in grade 3) (0 in gr. 3, 1 in gr. 4) 2 (0 in gr. 3, 1 in gr. 4)
Summative Assessment
English Language Arts Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
High Quality 2 2
Assessment (Reading) 25 (1 in gr. 3 & 4) (1 in gr. 3 & 4) 2 1
High Quality 2 2
Assessment (Writing)* 10 (1 in gr. 3 & 4) (1 in gr. 3 & 4) 2 0
Summative Assessment Design
Item Counts
Item Counts
11
High Quality Interim Assessment Design
Interim Assessment
Mathematics Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
High Quality 1 1
Assessment 25 2 (0 in grade 3) 1 (0 in grade 3)
Interim Assessment
English Language Arts Multiple Choice
Short Constructed Response
Extended Constructed Response
Performance Event Performance Task
High Quality
Assessment 25 2 1 1 1
Item Counts
Item Counts
Interim Assessment Design
12
Development and Administration
Costs Typical Assessment Program - $20/Student
Cost per student for the current typical assessment calculated using the ASG cost model (includes Year 0 development costs)
HQA Summative Assessment Program - $55/Student Cost per student for the high quality assessment calculated assuming
a single state implementation and no cost reduction strategies (includes Year 0 development costs)
Most states cannot afford a nearly tripling of their state assessment costs, so the result will be a limit on innovative assessment types, unless something occurs
13
Cost Reduction Strategies
Participation in a consortium Model includes state consortium sizes of 10, 20, and 30 states Use of a state consortium reduces costs by an average of $15 per student Consortium approach represents a significant decrease in assessment cost
Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items
Together, these innovations account for cost savings of about $3 to $4 per student
Likely to account for more as efficiencies are developed in programming and using technology for these purposes
Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring
14
Cost Reduction Strategies
Two different models for teacher-moderated scoring: Professional development model - no additional teacher compensation
beyond that supported by the state or district for normal professional development days
Stipend model - assume a $125/day stipend for teachers to score the performance items.
• Note: teachers were assumed to score all performance items in a distributed scoring model
These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid)
Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21
15
Overall Cost Reduction Results
16
Impact of Teacher Scoring Time on Costs
17
Cost Study Conclusions
The development cost of a new HQA is relatively inexpensive relative to the total cost of the assessment
A key factor in determining whether states can adopt and sustain new improved assessments is ongoing administration costs
In order to reduce costs, states should participate in an assessment consortium to share the overhead associated with development, administration, and management of assessments
Larger consortia are somewhat more cost-effective The majority of cost savings relative to the single state case are seen even in at
a 10-state consortium size States should strongly consider being part of a large consortium where certain
costs can be shared across many states, such as for item development and project management
18
Cost Study Conclusions
Implementing a HQA system with performance items is affordable, with teacher scoring of performance items at a price comparable to today’s assessments, when procured by a consortium of states
In order to implement and afford an HQA system that includes a variety of performance items, it is essential to have teachers involved in the scoring process
The cost impact of increases in the time to score performance items is very significant
The use of online technology (i.e., online assessments) should be encouraged
It has the potential to reduce assessment cost and improve quality The procurement of PCs to improve the student- to-PC ratio should be
encouraged at all levels of the educational system
19
Recommendations
Developing and implementing an HQA will likely cost more than most current state assessments. It can be affordable for states if they look carefully at
the design of the summative assessment component finding a balance in the number of CR items, PEs, and PTs used consider various cost-reduction strategies
It is recommended that state consortia go about the process of designing a new assessment in a thoughtful manner, then use a comprehensive costing model to analyze and determine the price in advance of any new assessment system they would like to implement
State consortia interested in implementing a HQA should make sure they can afford the ongoing administration costs of the assessment before they embark on developing it
20
For More Information
Edward Roeber
Michigan State University
263 Erickson Hall
East Lansing, MI 48824
(517) 432-0427
For a copy of the complete paper for this study, go to the
Assessment Solutions Group website to download a copy:
www.assessmentgroup.org