reciprocal accountability: using local and state assessment data to produce annual accountability...
TRANSCRIPT
![Page 1: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/1.jpg)
Reciprocal accountability: Using local and state assessment data to produce annual
accountability determinations
Scott MarionNational Center for the Improvement of Educational Assessment
CCSSO: National Conference on Student AssessmentSan Diego, CA
June 22-24, 2015
![Page 2: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/2.jpg)
2Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Beginning to Address Challenges of Assessment of CBE
![Page 3: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/3.jpg)
3Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comprehensive Assessment Systems
• When done well, competency-based assessment systems can be instantiations of comprehensive assessment systems– Multiple types of assessments, e.g., summative, interim, unit-
based, formative
– Multiple loci of control—classroom, school, district, and state• What is the role of local curriculum (not standards) in CBE-based
assessment design?
• In many (most?) cases, comprehensive assessment systems require state and local cooperation
• PACE’s reciprocal system requires tight coherence among the state, collaborative, and local assessment systems
![Page 4: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/4.jpg)
4Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Key Technical Challenges/Requirements for PACE
1. Reporting achievement (status)– In the language of USED, “annual determinations”
2. Calculating and reporting “growth”3. Defining, evaluating, and monitoring
comparability4. Ensuring and enhancing equity
• Focus today on producing comparable, annual determinations…
![Page 5: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/5.jpg)
5Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comparability
• A lot of energy focused on comparability over the past five years (e.g., PARCC-Smarter Balanced)
• Psychometricians worry a lot about the differences among things like equating, concordance, projection…– Congressional staff now worried about differences among “equal,”
“equivalent,” and “comparable”
• The U.S. has a particular obsession with “interchangeability,” but we always need to ask ourselves comparable for what
• We also need to learn from other countries and some enlightened measurement specialists in the US
![Page 6: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/6.jpg)
6Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comparability Challenges in a Reciprocal SystemPACE
Comparable Annual DeterminationsPACE Common Performance Task
District-Level Competency Scores
Competency 1
Local performance assessments
Competency 2
Local performance assessments
Competency 3
Local performance assessments
And so on
Smarter Balanced in three grades
![Page 7: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/7.jpg)
7Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Focus on Student Work
• Ultimately, comparability will be evaluated and achieved through intense interrogation of student work
• Multiple layers of the process– Within school calibration and comparable scoring– Development of common PLDs closely linked to Smarter Balanced– Across district comparability analyses– “Standard setting”
![Page 8: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/8.jpg)
8Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Cross-District Comparability
• Student work samples are used to evaluate the comparability of scoring (and, by inference, expectations) across districts– The goal is not to repeat the inter-rater reliability analyses conducted
at the local levels
• Rather, the plan is to evaluate prototypical and borderline papers from each district to judge the extent to which students are held to common expectations
• For example….
![Page 9: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/9.jpg)
9Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Matching Student Work to PLDs
Beginning Developing Approaching Target On Target Above Target Conceputalization
Equipartitions manipulatives of odd and even numbers (e.g., 27 divided in 3 groups; 24 divided in 4 groups) and explains equal or fair shares
places manipulatives within 100 into arrayed groups of more than two and explains (verbally or in written form) the relationship between repeated addition and multiplication
Self-creates models (e.g., manipulatives or drawings) within 100 into arrayed groups more than two to demonstrate the relationship between addition and multiplication
places manipulatives within 100 into arrayed groups of more than two and explains (verbally or in written form) the relationship between multiplication and division
Models and explains how models can show the relationship between multiplication and division
![Page 10: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/10.jpg)
10Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Standard Setting
• Initial achievement levels need to be established within-district– Unique assessments for determining competency
• Comparability analyses will help calibrate differences in standards (e.g., proficiency) across districts
• Profile approach will be used with the student work generated from the common tasks as anchors– Smarter Balanced results will be used as key benchmark data– Next year, we will have work samples collected throughout the year
![Page 11: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/11.jpg)
11Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Some additional resources
Domaleski, C., Gong, B., Hess, K., Marion, S., Curl, C., Peltzman, A. (2015). Assessment to support competency-based pathways. Washington, DC: Achieve. www.Achieve.org and www.nciea.org
Marion, S.F. & Buckley, K. (in press). Design and implementation considerations of performance-based and authentic assessments for use in accountability systems. In Braun, H. (ed). Meeting the Challenges to Measurement in an Era of Accountability. Washington, DC: NCME.
Marion, S. (2015, Feb). Two sides of the same coin: Competency based education and Student Learning Objectives. Published by Competency Works. http://www.competencyworks.org/resources/two-sides-of-the-same-coin-competency-based-education-and-student-learning-objectives/
Marion, S., & Leather, P. (2015). Assessment and accountability to support meaningful learning. Education Policy Analysis Archives, 23(9). http://dx.doi.org/10.14507/epaa.v23.1984
McCLarty, K. L. & Gaertner, M. (2015). Measuring mastery: Best practices for assessment in competency-based education. Washington, DC: American Enterprise Institute
![Page 12: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/12.jpg)
For more information:
Center for Assessmentwww.nciea.org
12Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Scott [email protected]
![Page 13: Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement](https://reader030.vdocument.in/reader030/viewer/2022032607/56649ed35503460f94be3a51/html5/thumbnails/13.jpg)
13Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Discussion Questions
1. What are the opportunities and challenges associated with implementing reciprocal accountability in your state?– Technical?– Policy?– Practical?
2. What should be the state’s role in a reciprocal system for ensuring that all students are being held to comparable expectations and being provided equitable opportunities to learn?