inter-rater reliability training - act holistic writing rubric
Post on 26-May-2015
352 Views
Preview:
DESCRIPTION
TRANSCRIPT
Inter-rater Reliability Training and Consensus Building
Peggy Muehlenkamp December 2012
Agenda● Background Building● Inter-rater Reliability/Consensus
Building● Scoring
○ plan○ rubric○ procedure
College/CareerReady
Writing on Demand
Inter-rater Reliability
Scoring vs. GradingScore(s): The number(s) or letter(s) assigned to an assessment via the process of measurement.
Grade(s): The number(s) or letter(s) reported at the end of a set period of time as a summary statement of evaluations made of students.
Transforming Classroom Grading, Robert Marzano
Holistic vs. Analytic
Procedure
● Papers are mixed: 6-8● 2 Readers will score each paper● Scores are averaged to determine overall
score
ExampleReader 1: 75 Score: MReader 2: 72 Score: T
Overall Score: _____
Ground Rules● Professional discussions
● Stay focused on task
● Refrain from conversations
● All scoring takes place in this room.
Scoring Tips● Know your Biases
● Let go of the Number Labels
● Take Comfort in the 2nd
Reader
Scoring Process
● Record Reader # on Paper
● Read Paper - quick, but thorough
● Score - overall impression
1st Decision
Example Prompt
Essay #1● clear understanding of the writing task ● demonstrates effective skill in responding to the
prompt.● takes a position and offers a critical context for
discussing the issue● essay addresses complexity by fully responding
to several counterarguments ● development of ideas is logical and specific
● the writer’s concise style and critical insight result in an essay that convincingly supports the main ideas as well as implies connection to further issues
● organization of the essay is clear● introduction and conclusion are clear and effective● essay demonstrates a good command of language,
using a variety of sentence constructions as well as varied and precise word choice
● few errors to distract the reader
Essay #1 (cont.)
Essay #2ACT Score: 2
● weak response to the task
● takes a position on the issue with a qualification
● thinly developed - very little explanation to
support each position
● some organization is evident
Essay #2 (cont.)
● supporting ideas are not grouped
● few transitions
● introduction and conclusion are not developed
● language use is simple
● little variety in sentence structure
● errors are sometimes distracting, but do not affect
understanding
Essay #3ACT Score: 1
● shows little skill in responding to the writing task
● takes a position, but fails to support with reasons
● minimal development of ideas
● some supporting claims are not understandable
Essay #3 (cont.)
● no discernable organization○ minimal introductory statement/no conclusion○ ideas are not logically grouped○ no transitions
● simple sentence structure and word choice
● language usage errors are distracting and make
some portions difficult to understand
Essay #4ACT Score: 2
● inconsistent skill in responding to task● takes a position/no development of counter-
argument● thinly developed - little explanation to support
second idea● some organization is evident
Essay #4 (cont.)
● discernible introduction
● conclusion is minimal
● simple transitions
● some variety in sentence structure
● appropriate word choice
Essay #5ACT Score: 4
● adequate response to the writing task● takes a position on the issue● shows recognition of complexity by
acknowledging counterargument● does not provide response to counter arguement● adequate development of ideas
○ ideas are developed with some specific reason and detail○ examples support ideas
Essay #5 (cont.)
● organization is clear
● somewhat developed introduction and conclusion
● obvious transitions throughout essay
● some logical sequencing of ideas is evident
● language use shows some sentence variety
● appropriate word choice
● errors do not impede understanding
Norming
Are We in Agreement?
top related