designing rubrics - learning in...
TRANSCRIPT
DESIGNING RUBRICS
Assessment Certificate Program
Workshop at DePaul University
March 25, 2015
By Ranfen Li, PhD
Acknowledgement
• The content of this presentation is based on
the following sources: • Myford, C. (Spring 2006), Performance Assessment, a
PowerPoint presentation
• Arter, J. and McTighe, J. (2001). Scoring rubrics in the classroom, R. Guskey & R.J. Marzano, Eds., Thousand Oaks, CA: Corwin Press
• Airasian, P and Russell, M.K(2008). Classroom assessment: concepts and application, New York, NY: McGraw-Hill
Objectives for this workshop
• Participants will be able to:
• Define what is a rubric and explain its benefits
• Identify situations in which rubrics are useful
• Gain knowledge of several types of rubrics
• Obtain skills in designing rubrics
Assessment
Gather information
• Testing – a formal, systematic procedure (assessment method)
Synthesize information
• Measurement – quantifying or assigning a number (scoring)
Interpret information
• Evaluation – judging quality or value
Assessment Methods
Major categories Selection and Supply – multiple
choice, short answer, concept map
Product – essay, paper, report, portfolio, thesis
Performance – experiment, musical performance, debate, presentation, behaviors
Scoring Methods
Major types
Assign points based on correct answers for selection or supply test items
Assign points based on a rubric for a performance-or product-based testing
A Rubric
• Is a scoring guide
•Has a set of clear expectations or
criteria, with all level score points
described and defined
Benefits of a rubric
• Increase the consistency of
judgments across teachers
and students, reducing
subjectivity of the
judgments
• Help instructors and
students focus on what is
valued in a subject, topic,
or activity
Relationship between Outcomes,
Assessment & Scoring Methods
Learning Outcomes
Assessment Methods
Scoring Methods
Situations When a Rubric Is Needed
Look for learning outcomes that:
• are skill based
• focus on actual performance of tasks in realistic settings
• would likely result in a product to be evaluated
• focus on students’ affective characteristics (i.e., feelings, values, attitudes, beliefs)
Examples of Outcomes
DePaul University undergraduate music students will be able to demonstrate the ability to perform with an acceptable tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness.
(source: one of the learning outcomes of B.M. in Music Performance at
DePaul University)
Examples of Outcomes
Apply nursing concepts, theories, and research to a family-centered and community based practice of professional nursing care and case management
(source: one of the learning outcomes of B.S. Registered Nurse at
DePaul University)
Examples of Outcomes
Students will be able to write analytical essays presenting an argument about one or more literary works and supporting it with appropriate evidence.
(source: one of the learning outcomes of B.A. in English at
DePaul University)
Examples of Outcomes
Students who engage with crisis management services at UCS will gain a new viewpoint on their problems and engage in perspective taking, thereby reducing potential for self-harm and self-sabotaging behaviors.
(source: an activity outcome of University Counseling Services at
DePaul University)
Introducing Rubrics and Other
Scoring Types
Rubrics Holistic vs. Analytic
General vs. Task Specific
Developmental
Other Types
Checklist
Rating Scales
Holistic Rubric
• A single rubric that
incorporates all the
performance criteria
• give a single score or
rating for an entire
product or performance
• rating is based on an
overall impression of a
student’s work
• Used for summative
assessment
Analytic Rubric
• Devise a separate rubric
for each performance
criterion and judge each
individually
• give a separate score
for each criterion
• Reveal a profile of
strengths and
weaknesses
• Used for formative
assessment
Example for a Holistic Rubric
Outcome: Demonstrate the ability to perform with an
acceptable tone quality, pitch and rhythmic accuracy,
dynamic control, articulation, and expressiveness.
Level 3 - Acceptable
Demonstrate excellent performance in all of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness
Level 2 - Partially Acceptable
Demonstrate good performance in most of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness
Level 1 - Not acceptable
Show poor performance in most of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness
Holistic Rubric - Advantages
• Useful for getting a “quick snapshot”
of overall quality or achievement
• More cost efficient; can be carried out
relatively quickly to score large
numbers of responses
• Useful when a single score is needed
Holistic Rubric - Disadvantages
• No detailed analysis of strengths and
weaknesses of a product or
performance
• Don’t provide students with detailed
feedback to guide their improvement
• Two students can receive the same
score for different reasons
Example for an Analytic Rubric
Outcome: Demonstrate the ability to perform with an
acceptable tone quality, pitch and rhythmic accuracy,
dynamic control, articulation, and expressiveness. Traits Not Acceptable
(Level 1)
Partially
Acceptable
(Level 2)
Acceptable
(Level 3)
Tone Quality Description for
level 1
Description for
level 2
Description for
level 3
Pitch/Rhythmic
Accuracy
Description for
level 1
Description for
level 2
Description for
level 3
Dynamic Control Description for
level 1
Description for
level 2
Description for
level 3
Articulation Description for
level 1
Description for
level 2
Description for
level 3
Expressiveness Description for
level 1
Description for
level 2
Description for
level 3
Analytic Rubric - Advantages
• Useful for judging complex performances or products involving several significant dimensions
• Provide more specific information or feedback to students about strengths and weaknesses of a performance or product
• Help students to better understand the nature of quality work
Analytic Rubric - Disadvantages
• Often slower to score (requires
giving ratings on multiple
dimensions)
• May take longer to learn to apply
• May be more expensive to use; not
as good a choice for large-scale
assessments
Generic Rubric
• Can be used for multiple tasks across similar performances or products. Useful when:
• skills being assessed are complex and generalize across tasks
• the goal is to help students see what quality looks like across similar tasks
• students will not all be doing exactly the same task
Task-specific Rubric
• Can only be used for a single task. Each task has its own rubric. Useful when you want to know whether students have learned:
• how to carry out a specific task, perform a particular skill, or produce a particular product
• particular facts, equations, methods, or procedures
Developmental Rubric
• Used to monitor development of a process, product, behavior, etc.
• Useful when the goal of evaluation is to determine level of development
• Rubric can be based on relevant developmental theory
• Usually used several times to examine changes over a period of time
Writing Rubrics
Steps Identify performance criteria
Determine the number of scoring points
Write performance level descriptions
Identify Performance Criteria
• List important aspects (elements, behaviors, components, qualities, features, characteristics) of the performance (or product) students will demonstrate. Ask yourself:
• What do you want your students to demonstrate from carrying out this process or producing this product?
• What would a high quality product look like? What are its essential, defining characteristics or features?
Determine the Number of Score Points
• Ask yourself, “How many points do I need to
adequately describe the range of performance I am
seeing in student work?”
• “Normal” range of points is 3 to 7
• If you have samples of student work (e.g., essay),
sort the samples into three or four groups based on
level of quality (e.g., well written, fair, poor).
- Write down reasons for placing pieces into piles.
- Cluster the reasons into important dimensions of
performance.
Write Performance Level Descriptions
• Start by describing the highest and lowest levels of quality, and then fill in the middle levels.
• Use clear language, defining terms for student-friendly language.
• Avoid making the rubrics too lengthy and laden with details.
Checklist
• Is a written list of performance criteria
• Scorer makes two-category judgment of each
performance criteria included in the checklist
• Examples of Two-Category Judgments
• Yes/No
• Present/Absent
• Performed/Not Performed
• Met/Not Met
• Observed/Not Observed
Example for Checklist
Outcome: Students will be able to design and carry out
chemical experiments, record data and analyze results.
• Direction: The instructor would place a checkmark before
each criterion that was met (and leave blank those criteria
that were not met).
_____ Design a chemical experiment
_____ Conduct the experiment
_____ Record experiment data
_____ Analyze the data
_____ Interpret the results
(source: one of the learning outcomes of B.S. in Chemistry at DePaul University)
Alternative Format for Checklist
Tasks Performed Not performed
Design a chemical
experiment Conduct the
experiment Record experiment
data Analyze the data
Interpret the results
• There are other formats that will work, as well.
• Use whatever format that seems to work.
Rating Scales
• Allow the observer to judge
performance along a continuum rather
than as a dichotomy
• Three commonly used types
• Numerical
• Graphic
• Descriptive graphic
Characteristics of Numerical Rating
Scales
• Each number signifies a point on the scale.
• The teacher circles a number to make a
judgment about student performance with
respect to that performance criterion.
• Each number has a verbal description that
remains constant from one performance
criterion to the next.
Example of Numerical Rating Scales
• Direction: Indicate the degree to which this student carries out
chemical experiments by circling the appropriate number. The
number represents the following rules: 3 = performs very well, 2
= performs well, and 1 = performs poorly.
Carry out chemical experiments
1 2 3
Characteristics of Graphic Rating
Scales • Each performance criterion is followed by a
horizontal line.
• The teacher makes a check on the line at
the point that best describes the student’s
product (or performance).
• A set of categories identifies specific
positions along the line, but the teacher is
free to check between these points.
Example of Graphic Rating Scales
• Direction: Indicate the degree to which this student carries out
chemical experiments by placing an X anywhere along the horizontal
line under the criterion.
To what extent does the student carry out chemical experiments?
Very well Well Poorly
Characteristics of Descriptive
Graphic Rating Scales
• Use descriptive phrases to identify the points
on a graphic rating scale.
• The descriptions are thumbnail sketches of
how students behave at different points along
the continuum (or how the quality of a
product differs).
• In some scales, only the center and end
points are defined. In others, a descriptive
phrase will appear beneath each point.
Example of Descriptive Graphic
Rating Scales • Direction: Mark your rating by placing an X anywhere along the
horizontal line under the criterion.
To what extent does the student follow the safety
procedures while carrying out chemical experiments?
Consistently follows
Generally follows but needs
improvement
Sometimes follows and needs
improvement Never follows
Time to practice
1. Write one learning outcome that would result in
a product or a performance based assessment
2. List a few observable performance criteria that
you will use to evaluate the students’
performances or products
3. Decide what type of an assessment rubric or a
rating scale you will build
4. Determine the number of scoring points
5. Sketch out descriptions of different scoring
levels