objective observations: creating rubrics for performance

© 2010 National Board of Osteopathic Medical Examiners www.nbome.org

OBJECTIVE OBSERVATIONS:CREATING RUBRICS FOR PERFORMANCE ASSESSMENTLaurie Gallagher D.O. Chief Physician Trainer

Jeanne Sandella D.O. Vice President for Clinical Skills Testing

AACOM/AODME

April 24, 2015

© 2015 NBOME

www.nbome.org

Rubric “Workshop”

• Background on Using Rubrics 10 mins

• Rubric Development 10 mins

• Apply Ratings 5 mins

• Questions 5 mins

Objectives

© 2015 NBOME

www.nbome.org

Challenge in Education

• Provide an objective measurement of learner behavior for various purposes

• Formative vs Summative

• Checklists vs Rubrics

© 2015 NBOME

www.nbome.org

What is a Rubric ?

• Scoring tool: Detailed scoring guide

• Coherent set of criteria to assess performance

• Includes descriptions of quality of performance

© 2015 NBOME

www.nbome.org

Why Use A Rubric ?

• Scoring is objective

• Match level of performance, not judge it

• Based on pre-set expectations and descriptions

• Reproducible outcomes

• Reusable

• Time efficient

© 2010 National Board of Osteopathic Medical Examiners www.nbome.org

Excellent Good SatisfactoryNeeds

Improvement

Componentsof the Report

Question/Purpose

Spelling, Punctuation,

Grammar

.

PERFORMANCE RATING

PERFORMANCE DESCRIPTIONS

CR

ITER

IA

© 2015 NBOME

www.nbome.org

Commonly used Rubrics

• Analytic: each criterion is evaluated independently

• Holistic: all criteria are evaluated simultaneously

© 2015 NBOME

www.nbome.org

Advantages

Analytic:• Good for formative

assessment

• Can link to instruction

• Gives feedback to student and instructor/rater

• Two columns of bulleted text

Holistic:• Good for summative

assessment

• Scoring is fast

• Achieve inter-rater reliability in a shorter time frame

• Two columns of bulleted text

© 2015 NBOME

www.nbome.org

Beginning1

Developing2

Accomplished3

Exemplary4

Score

Criteria1

Description reflectingBeginning level of performance

Description reflectingmovement towards masterylevel of performance

Description reflectingachievement of masterylevel of performance

Descriptionreflectinghighest level of performance

Criteria2

DescriptionreflectingBeginning level of performance




Criteria3





Criteria4





Analytic Scoring Rubric

© 2015 NBOME

www.nbome.org

Holistic Rubric

Score Description

5 Demonstrates complete understanding of the problem. Flawless grammar

4 Demonstrates considerable understanding of the problem.An occasional grammar error.

3 Demonstrates partial understanding of the problem. Few grammatical errors.

2 Demonstrates little understanding of the problem. Some grammatical errors

1 Demonstrates no understanding of the problem. Numerous grammatical errors

0 No Response/task not attempted

© 2015 NBOME

www.nbome.org

Challenges to Rubric Creation

• Concrete criterion examples

• Creating clear outcome measures

• The rubric is only as good as the preset definitions or descriptors

© 2015 NBOME

www.nbome.org

Benefits of a Well Constructed Rubric

Creates measurable outcomes for performance assessment

Creates reproducible outcomes for performance assessment

© 2015 NBOME

www.nbome.org

Rubric Websites and References

• http://Rubistar.4teachers.org

• www.teachnology.com• References:• Hall Wikfors E, Salmon, S, Chocolate chip cookies and rubrics, helping students understand

rubrics in inclusive settings. Teaching Exceptional Children. 2003; 4:8-11.

• Jamieson S. Likert Scales: How to (ab)use them. Med Educ. 2004; 38: 1217-8.

• McLeod, S.A. Likert Scale. Retrieved from http://www.simplypsychology.org/likert-

• Morton J. Performance-based assessment in undergraduate medical education. The Clinical Teacher 2007;4: 36-41.

•

• Whelan GP, Boulet JR, McKinley DW. Scoring standardized patient exams: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Med Teach. 2005;(3):200-6.

http://rubistar.4teachers.org/

http://www.teachnology.com/

© 2015 NBOME

www.nbome.org

Crayon Rubric

Descriptors Descriptors Descriptors

Criterion 1

Criterion 2

Criterion 3

Criterion 4

© 2015 NBOME

www.nbome.org

Crayon Rubric

Unacceptable Satisfactory Superior

Color Quality

Line Sharpness

Fill Pattern

Texture

© 2015 NBOME

www.nbome.org

Crayon Rubric

Unacceptable Satisfactory Superior

Color Quality • Dull or muted• Not consistent

• Bright/intense• Consistent

Line Sharpness • Point dulls quickly• Thick• Edges blurred

• Point stays sharp• Thin• Crisp edge

Fill Pattern • Skip patterns/irregular• Some areas thicker • Difficult to apply

• No white gaps on paper• Even fill thickness• Easy to apply

Texture • Sticky/tacky• Doesn’t adhere to paper

• Silky smooth• Stays on paper

© 2015 NBOME

www.nbome.org

Pitfalls

• Complicated/ambiguous performance descriptors

• Irrelevant criteria

• Bias

• Drift

• Inadequate training

objective observations: creating rubrics for performance

Documents