Deakin University CRICOS Provider Code: 00113Bdeakin.edu.au/learning/enhancing-your-practice/cradle
The ‘unknown unknowns’ of assessment rubrics in practice, policy and research
Associate Professor Phillip DawsonAssociate Director
Centre for Research in Assessment & Digital Learning (CRADLE)Deakin University, Melbourne, Australia
@phillipdawson
Deakin University CRICOS Provider Code: 00113B
Deakin University CRICOS Provider Code: 00113B
A couple of decades ago, rubric began to take on a new meaning among educators. Measurement specialists who scored students’ written compositions began to use the term to describe the rules that guided their scoring. They could have easily employed a more readily comprehensible descriptor, such as scoring guide, but scoring guide lacked adequate opacity. Rubric was a decisively more opaque, hence technically attractive, descriptor. (Popham 1997, p. 72)
Deakin University CRICOS Provider Code: 00113B
Hands up if rubrics make you feel:• Confident• Concerned• Relieved
• Confused• Excited• Bored• …another feeling
you’d like to share?
Deakin University CRICOS Provider Code: 00113B
Pre-1997
19982000
20022004
20062008
20102012
20140
1000
2000
3000
4000
5000
6000
7000
Mentions on Google ScholarMentions in books
Cumulative frequency, 28 April, 2015.
Deakin University CRICOS Provider Code: 00113B
What is a rubric?• A tool used in the process of evaluating
student work, which usually includes:– evaluative criteria– quality definitions for those criteria at
particular levels – a scoring strategy.
Per Popham (1997)
Deakin University CRICOS Provider Code: 00113B
This workshop• Overview of framework• Practice sharing• Scenarios
Deakin University CRICOS Provider Code: 00113B
Framework overview• 14 ‘design elements’ representing choices
made in rubric design• Brief overview of each – stop me if you
want more detail
Deakin University CRICOS Provider Code: 00113B
Specificity: the particular object of assessment
• Generic– Including faculty-wide
or course level• Task-type• Task-specific
Deakin University CRICOS Provider Code: 00113B
Secrecy: who the rubric is shared with, and when it is shared
• Top-secret rubrics• Rubrics returned with marks/feedback• Rubrics shared with task instructions
Deakin University CRICOS Provider Code: 00113B
Exemplars: work samples provided to illustrate quality
What does a good one of these look like?
Why is this one good?
Deakin University CRICOS Provider Code: 00113B
Scoring strategy: procedures used to arrive at marks and grades
Analytic• Adding up• Logic, conditions,
thresholds• Combining scores from
multiple markers
Holistic• Assessor uses all
information to make a single overall quality judgement
Deakin University CRICOS Provider Code: 00113B
Evaluative criteria: overall attributes required of the student
• (usually the leftmost column)• Mandatory (or not?)• Connector between indicators of quality
and task/subject outcomes
Deakin University CRICOS Provider Code: 00113B
Quality levels: the number and type of levels of quality
• Grade levels (P/C/D/HD)
• Learning outcomes taxonomies
• Levels of study
• Fail?• Every criterion at
every level?
Deakin University CRICOS Provider Code: 00113B
Quality definitions: explanations of attributes of different levels of quality
“Mixture of qualitative and
quantitative feedback sources”
Deakin University CRICOS Provider Code: 00113B
Judgement complexity: the evaluative expertise required of users of the rubric
Pass Credit Distinction
Clarity of expression
Bad Acceptable Good
Deakin University CRICOS Provider Code: 00113B
Judgement complexity: the evaluative expertise required of users of the rubric
Pass Credit Distinction
Clarity of expression
Five or fewer grammatical errors
Two or fewer errors
Zero grammatical errors
Deakin University CRICOS Provider Code: 00113B
Users and uses: who makes use of the rubric, and to what end
Teachers• Marking• Providing feedback• Communicating
standards• Understanding standards
Students• Planning work• Understanding standards• Self assessment• Peer assessment• Grade grubbing• Planning improvement to
future work
Deakin University CRICOS Provider Code: 00113B
Creators: the designers of the rubric
• Teachers• Textbook publishers• Internet
communities• Students
• Rubric specialists• Researchers• Professional bodies
Deakin University CRICOS Provider Code: 00113B
Quality processes: approaches to ensure the reliability and validity of the rubric
• Formal reliability and validity processes• Informal review, piloting and revision• Haven’t got time for that, just see how it
goes
Deakin University CRICOS Provider Code: 00113B
Accompanying feedback information: comments, annotation or other notes on student performance
• Stimulus for feedback discussion• Focal point for written feedback• A supplement to written feedback• Replacement for written feedback
Deakin University CRICOS Provider Code: 00113B
Presentation: how the information in the rubric is displayed
Deakin University CRICOS Provider Code: 00113B
Explanation: instructions or other additional information provided to users
• Written instructions• Discussion around rubric in class• Marking workshop• Video explanation
Deakin University CRICOS Provider Code: 00113B
How do you use rubrics?• Do you feel strongly about any of these
elements?• Discuss how you use rubrics in groups of
three or four• Structure your conversation around the
framework
Deakin University CRICOS Provider Code: 00113B
Scenarios – choose one, in groupsRubric advice• 1st year educational
theory course• 1000 students• Lecturer wants your
advice on rubric for 2000 word essay
Faculty rubric policy/procedures• Education faculty at a
large university have contracted you to write rubric policy/procedures
• What would you mandate?
Deakin University CRICOS Provider Code: 00113B
Questions/Comments?Pre-
19971998
20002002
20042006
20082010
20122014
0
1000
2000
3000
4000
5000
6000
7000
Mentions on Google ScholarMentions in books
@phillipdawson