clarifying fuzziness: writing rubrics for direct and ...apps.naspa.org/cfp/uploads/naspa rubric...
TRANSCRIPT
3/15/2017
1
Clarifying Fuzziness:
Writing Rubrics for Direct and Indirect Assessment2017 NASPA Annual Conference March 13, 2017 – San Antonio
Dr. Erica EckertKent State University
Dr. Brenda McKenzieVanderbilt University
(C) E. Eckert & B. McKenzie, 2017
Your Presenters
ASSESSMENT, TECHNOLOGY
Dr. Erica Eckert Assistant Dean for
Assessment and Accreditation, Assistant Professor, Kent State University
5 years in academic assessment and accreditation
5 years in admissions operations and systems/technology
STUDENT AFFAIRS, FACULTY
Dr. Brenda McKenzie
Senior Lecturer, Higher Education Administration, Vanderbilt University
21 Years of experience in Student Affairs (Leadership education, student activities, orientation, residence life)
2 years faculty in graduate preparation programs
(C) E. Eckert & B. McKenzie, 2017
3/15/2017
2
Our Goals for You
…identify several types of rubrics
…apply the concept of blueprinting to the
organization of checklist and performance-level
rubrics
…evaluate opportunities to deploy rubrics of
varying types
…participants will design descriptive rubrics for a
provided scenario
(C) E. Eckert & B. McKenzie, 2017
Types of Assessment Evidence
DIRECT
Evidence of student
learning is tangible or
visible, self-
explanatory;
demonstrates what
students have and
have not learned
INDIRECT
Evidence consists of
signs that students are
probably learning,
can include reflection
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, p. 20)
3/15/2017
3
Selecting Methods and
Approaches
Criteria for choosing methods
Content of tool
Quality/consistency of items (reliability)
Focus/truth of items (validity)
Feasibility (time, cost)
Motivation
Other Considerations (fairness, format, interpretability,
consequences to participants)
(C) E. Eckert & B. McKenzie, 2017
(Banta & Palomba, 2015, pp. 74-79)
Evaluating Values, Dispositions,
Habits of the Mind
Usually indirectly assessed
These are hard to measure… focus on how they
are expressed
2 Methods: Reflection, Behaviors
Suggested techniques: surveys, minute papers,
before-and-after reflection, journals, interviews,
rating scales… several of which can rely on
rubrics
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 183-201)
3/15/2017
4
Rubric Basics
Rubric – A scoring guide or grading criteria,
typically a list or chart, crafted to evaluate
assignments
Focus on elements/items you are “looking for”
The items may be accompanied by guidelines as to the
evaluation criteria
“There is no single correct way to write or format rubrics”
(p. 138) assuming one keeps in mind the rubric’s
purpose
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 137-138)
Activity: Blueprinting
Elements for a Rubric
How I love to evaluate thee, let me think about the ways…
(C) E. Eckert & B. McKenzie, 2017
3/15/2017
5
Formatting Options for Rubrics
Checklist Rubric Yes/No
Rating Scale Rubric Example: Poor, Fair, Good, Very Good
Holistic Rubrics Series of broad statements with levels (one rating)
Descriptive Rubric Performance levels are explained in detail
Structured Observation Guide Evaluation of focused behaviors/factors with text
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 138-148)
Checklist Rubric
• Contains a list of
tasks
• Evaluator notes
which elements are
present (or not)
• Pros/Cons?
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 138-139)
3/15/2017
6
Activity: Creating a
Checklist Rubric
Generate 5 checklist items for evaluation
(C) E. Eckert & B. McKenzie, 2017
Rating Scale
Rubric
• Contains a list of
elements or tasks
• Each element or
task is assigned a
rating (e.g., A, B, C,
D, F)
• Pros/Cons?
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 140-142)
3/15/2017
7
Holistic Rubric
• Complex
descriptions of a
combination of
performances/
behaviors resulting
in a single score
• Pros/Cons
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 145-146)
Descriptive
Rubric
• Contains a list of
elements or tasks
• Each element or
task has a series of
descriptive
performance levels
• Pros/Cons?
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 142-145)
3/15/2017
8
Effective Descriptive Rubrics…
Focus on the most important tasks
Are short; long rubrics can be confusing
Use concrete action verbs
Leave room for the student to display creativity
Are task/element oriented (rating scales second)
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 148-151)
Effective Descriptive Rubrics…
Use 3-5 performance levels
Label each level with names (not just numbers)
Use descriptive statements for performance levels
with meaningful variation
Are piloted before being used
(C) E. Eckert & B. McKenzie, 2017
(Suskie, 2009, pp. 148-151)
3/15/2017
9
Activity: Creating a
Descriptive Rubric
(C) E. Eckert & B. McKenzie, 2017
When would
you use a
checklist rubric?
(C) E. Eckert & B. McKenzie, 2017
3/15/2017
10
When would
you use a
holistic rubric?
(C) E. Eckert & B. McKenzie, 2017
When would
you use a rating
scale rubric?
(C) E. Eckert & B. McKenzie, 2017
3/15/2017
11
When would you
use a descriptive
rubric?
(C) E. Eckert & B. McKenzie, 2017
What makes a
rubric capable of
collecting directevidence?
(C) E. Eckert & B. McKenzie, 2017
3/15/2017
12
References, Questions
Banta, T. W., & Palomba, C. A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass.
Stevens, D.D., & Levi, A. J. (2013). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus.
Suskie, L. A. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.
(C) E. Eckert & B. McKenzie, 2017