collaborating with teachers for next generation item development
DESCRIPTION
Collaborating with Teachers for Next Generation Item Development. National Conference on Student Assessment June 29, 2012. Engaging teachers in the process of writing assessment items Overview of each state’s program Rationale for the approach involving teachers as item writers - PowerPoint PPT PresentationTRANSCRIPT
Collaborating with Teachers for Next Generation Item Development
National Conference on Student Assessment
June 29, 2012
• Engaging teachers in the process of writing assessment items Overview of each state’s program Rationale for the approach involving teachers
as item writers Use of technology in the development process Benefits including professional development Lessons learned and next steps Next-generation assessment systems
Presentation Focus
•John Moon, NeSA Project Director, Nebraska Department of Education
•Andrew Middlestead, Test Development Manager, Michigan Department of Education
•Patty McDivitt, Data Recognition Corporation
Presenters
John MoonNeSA Project DirectorStatewide Assessment Office [email protected]
Item Development by Nebraska Teachers
Overview of the Assessment Program:
Nebraska State Accountability (NeSA)
•Summative Assessment administered in Spring
•Grades 3-8 and 11 for Reading and Math•Grades 5, 8, and 11 for Science•Grades 4, 8, and 11 for Writing•Teacher Developed Items
Overview of the Assessment Program:
Check4Learning(C4L)
•Summative/Formative Assessment System administered throughout the year
•State-level Item Bank of Multiple Choice Items
•For Reading, Math, and Science – all grades
•Practice Tests for Writing – all grades•Teacher Developed Items
Classroom Assessment(C4L)
Interim/Benchmark Assessments (C4L)
Statewide Assessments (NeSA)
Aligned to State StandardsBalanced Assessment
System
•Both formative (C4L) and summative (C4L/NeSA) assessments are valuable and important
•Without both, the classroom assessment system is not balanced
BALANCED CLASSROOM ASSESSMENT SYSTEM
FORMATIVE (C4L) ASSESSMENTS
SUMMATIVE (NESA & C4L) ASSESSMENTS
A process used by teachers and students during instruction that provides immediate feedback to adjust ongoing teaching and learning to help students improve their achievement of intended instructional outcomes.
A tool used after instruction to measure student achievement which provides evidence of student competence or program effectiveness.
Overview of the Assessment Program Check4Learning: What is it?• State-level item bank
of
• multiple-choice items in
• reading, mathematics, and science that are
• stored in the C4L online system and
• used to generate online interim assessments to “wrap around” NeSA tests.
How can C4L be used?•Teachers, principals, or central office staff
can identify items to
•generate mini tests for use at point of instruction so that
•a check for learning can occur at
•any time throughout the year.
Organization of Item Writing
•Table of Specifications (TOS) – What’s needed? Examine indicators with highest number of
required items Set grade level goals for the item writing
sessions•Item Bank – What do we have?
Inventory items at each grade by indicator Identify indicators needing additional items Examine DOK and difficulty levels for additional
items
Rationale for the Approach:Work Session Item
Development – NeSA and C4L
•To understand a balanced assessment system
•To develop quality items for tested indicators
•To identify the characteristics of quality multiple-choice items
•To acquire adequate information to support others on the balanced assessment system
A Quality Test . . . • Accurately and
appropriately measures what is relevant.
• Is consistent and precise.
• Does not put any group at a disadvantage.
Validity
Fairness
Reliability
1. Assessments reflect the state standards.2. The students have had an opportunity to
learn the content.3. Assessments are free from bias or
offensive situations.4. Assessment levels are appropriate for
students.5. The mastery levels are appropriate.
Quality Assessment Criteria
Item Writing Guidelines
• Content• Style• Item Stems• Response Options
Item Writing Guidelines - CONTENT
• Should be clearly aligned to the identified indicator
• Should not reference material or information presented in another item
• Should focus on one idea or concept• Should test meaningful content
Item Writing Guidelines - STYLE
• Spell out acronyms• Minimize the use of “NOT” and
“EXCEPT”• Avoid absolutes (e.g., always, never)
and vague modifiers (e.g., best, worst)• Use appropriate vocabulary• Ensure correct grammar, punctuation,
capitalization, spelling• Use active voice and present tense
Item Writing Guidelines – ITEM STEMS
• State clearly and concisely• Use complete sentences with ending
punctuation in context• Question appears at the end of the stem• Use which before a noun and what before
a verb• Avoid negatives (capitalize if used)• Do NOT use “Which of the following . . .”
Cognitive Demand
• The kind and level of thinking required of students to successfully engage with and solve a task• Ways in which students interact with
content
Three Different DOK LevelsDOK 1- Describe three characteristics of
metamorphic rocks. (Requires simple recall)
DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types)
DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)
Item DifficultyDifficulty is a reference to how many students answer a question correctly.
“How many of you know the definition of exaggerate?” DOK 1 – recall If all of you know the definition, this question is an easy question.
“How many of you know the definition of prescient?” DOK 1 – recall If most of you do not know the definition, this question is a difficult question.
Item Review Check List
• Item Stem▫Content – alignment and focus▫ Style – clear and concise
• Responses – plausible and homogenous• Key – correct answer• DOK – appropriate and correct• Difficulty – appropriate and correct
Andrew MiddlesteadOffice of Educational Assessment and Accountability in the Michigan Department of [email protected]
Item Development by Michigan Teachers
Overview of the Assessment Program Michigan currently has 6 separate assessment
programs which it develops and administers Michigan Educational Assessment Program
(MEAP) Michigan Merit Examination (MME) MEAP-Access (2% population) MI-Access (1% population) English Language Proficiency
Assessment(ELPA) Michigan Interim Assessments (K-12)
Overview of the Assessment ProgramMichigan uses Michigan educatorsthroughout the development process of the
majority of test items that are needed for all
of these assessment programs.
Teacher-Written Items: Development Model Michigan educators:
Write items based on Michigan content expectations and any guidance from Michigan and DRC staff
Review all items through both a Content and Bias/Sensitivity review session
Review field-test data from both a content and bias/sensitivity perspective
Teacher-Written Items: Development Model (continued)Michigan educators:Michigan published children’s authors:
Write commissioned passages for use in our assessments.
Rationale for the Approach•Michigan has been using educators to
write and review assessment items for up to 10 years.
•Having educators involved heavily in the test development provides some sense of ownership of the Michigan assessments their students see. (though it doesn’t mean they like it )
• Items are only developed by state or contractor staff as a last resort in unique situations.
Use of Technology in the Teacher-Written Item Development Process
•Michigan has developed its own online Item Banking and Test Development system.
•Educators are able to work online in Item Writer Training sessions, or independently on tasks
•Tasks are able to be done quicker and more efficiently with far less paper (we use hardly any)
Use of Technology in the Teacher-Written Item Development Process
Benefits•The largest benefit I see in having Michigan educators be so involved in our item development process is:
They can go back to their school and talk about the process to hopefully increase educator buy-in of the assessment program.
Benefits (continued)•Benefits I see in having our item development online is:
Faster turn-around time from item writing, to reviewing, to consolidated comments.
Less paperConsistent with the times.
Lessons Learned•Importing items and their attributes from
various sources into one item bank isn’t so easy.
•Transition time to realize efficiencies•With such high educator involvement,
regularly evaluate your training materials and methods
Future Plans and Next Steps Goal of 100% commissioned passages
12 Michigan published children’s authors have been writing pieces across several assessment programs for the past 18 months
These authors are thrilled! Commissioned work has it’s benefits!
Enhancing Michigan’s Item Banking System To provide more flexibility in program and
item types Technology-based items
Two Models for Teacher-Developed Items Nebraska
Centralized training and item writing workshops; immediate feedback provided
Decentralized training is not an option
Teachers use a user-friendly MS Word template; items imported into the item bank
Michigan Centralized training
and item writing workshops; immediate feedback provided
Decentralized training is an option
Teachers receive training and write directly inside the item banking system
Two Models for Teacher-Developed Items Nebraska
External formatting of items for publication
Once the item is submitted and accepted, the involvement of the teacher in writing the item is done
Michigan Internal formatting
of items for publication
Once the item is submitted and accepted, the involvement of the teacher in writing the item is done
Role of the Vendor in Teacher-Developed Items•Seek to provide support to the process•Seek to ensure that the specific concerns
of assessment such as source of challenge are considered without losing the intent of the writer
•Seek to ensure that the items adhere to style specifications and pre-determined technical quality
Next-Generation Assessment Systems• Much as teachers have new and novel ways to
approach instruction in the classroom, teachers as writers will display creativity in addressing the standards in new and novel ways, including • becoming invested in the deep understanding of
the standards and the development of the assessments
• helping to promote the link between assessment and instruction in the classroom, and
• viewing assessments in a more positive light due to peer involvement in the process.
Next-Generation Assessment Systems
•Teachers will become familiar with the use of templates and/or the entire item-banking process (shared item components).
•Teachers will use what they have learned in the item-writing workshops to develop classroom-based assessments.
Next-Generation Assessment Systems Teachers will become familiar in the use of
item-writing guidelines and item specifications.
Teachers will gain understanding in the structure of items designed to measure the standards, including Evidence-based design items Selected-response items Technology-enhanced items Other
Next-Generation Assessment Systems Writer’s workshops (professional
development) Focus upon collaboration among teachers
and/or the department Engage teachers in direct writing;
providing regular and direct feedback Engage teachers is peer review Empower teachers with responsibility and
ownership of the development process
Questions and Answers