designing grading rubrics to assess student learning the diy (do it yourself) workshop mary t....

Designing Grading Designing Grading Rubrics to Assess Rubrics to Assess Student LearningStudent Learning

The DIY (Do It Yourself) The DIY (Do It Yourself) WorkshopWorkshop

Mary T. Blackinton PT, MS, EdDMary T. Blackinton PT, MS, EdDDirector Transition DPT ProgramDirector Transition DPT ProgramChair, PT Outcomes CommitteeChair, PT Outcomes CommitteeNova Southeastern UniversityNova Southeastern University

Dr. Mary Blackinton, 01-19-08

AgendaAgenda Introduction to GradingIntroduction to Grading

Myths & Principles of Effective GradingMyths & Principles of Effective Grading Choosing an Assignment for a RubricChoosing an Assignment for a Rubric

Analyzing Grading RubricsAnalyzing Grading Rubrics Small group discussionSmall group discussion

Developing RubricDeveloping Rubric DefinitionsDefinitions Primary Trait AnalysisPrimary Trait Analysis

Creating & Using RubricsCreating & Using Rubrics DesignDesign PitfallsPitfalls


ObjectivesObjectives Upon completion of the workshop, faculty will be Upon completion of the workshop, faculty will be

able toable to:: Describe the role of grading rubrics in: Describe the role of grading rubrics in:

Clarifying teacher expectationsClarifying teacher expectations Evaluating student learningEvaluating student learning Providing detailed feedbackProviding detailed feedback Improving instructionImproving instruction Performing outcome assessmentPerforming outcome assessment

Evaluate the strengths and weaknesses of grading rubrics Evaluate the strengths and weaknesses of grading rubrics as a student assessment tool in health professions as a student assessment tool in health professions educationeducation

Integrate the principles of primary trait analysis (PTA) into Integrate the principles of primary trait analysis (PTA) into designing a rubric for an assignment or performance skilldesigning a rubric for an assignment or performance skill

Create a grading rubric for an assignment, project, or Create a grading rubric for an assignment, project, or examexam


Grading Challenge # 214:

Student’s Perceptions


Why do we Grade?Why do we Grade? GradingGrading11

The The processprocess used by faculty to assess student used by faculty to assess student learning via assignments and exams, learning via assignments and exams, including:including:

Relating test items or assignments to learning Relating test items or assignments to learning objectivesobjectives

Establishing criteria/standards (SAFETY)Establishing criteria/standards (SAFETY) Helping students acquire needed knowledge/skillsHelping students acquire needed knowledge/skills Facilitating student motivationFacilitating student motivation Giving feedback about performanceGiving feedback about performance Communicating about what has been learned Communicating about what has been learned Using results to influence teaching and curriculumUsing results to influence teaching and curriculum

11 BE Walvoord, VJ Anderson (1998). BE Walvoord, VJ Anderson (1998). Effective grading: A Effective grading: A tool for learning and assessmenttool for learning and assessment. Jossey-Bass: San . Jossey-Bass: San FranciscoFrancisco


Student Assessment in Health Professions

Faculty charged to assess competency in patient Faculty charged to assess competency in patient carecare

Responsibility to protect publicResponsibility to protect public Other stakeholders: accrediting bodies, profession, students, Other stakeholders: accrediting bodies, profession, students,

parents…parents…

Professionals integrate 3 domains of learning Professionals integrate 3 domains of learning Cognitive-thinking/reasoningCognitive-thinking/reasoning Affective-feeling, valuing, ethicsAffective-feeling, valuing, ethics Psychomotor-doing, manipulating, performing, administeringPsychomotor-doing, manipulating, performing, administering

Assessment strategies reflect demands Assessment strategies reflect demands professional edprofessional ed

Must include more than multiple choice assessment to Must include more than multiple choice assessment to determine learningdetermine learning

Often have multiple faculty grading student performanceOften have multiple faculty grading student performance


Myths About GradingMyths About Grading

Grading:Grading: Is a necessary EVILIs a necessary EVIL Must be objectiveMust be objective Should be easyShould be easy Should not take detract from teachingShould not take detract from teaching Use knowledge of experts to evaluate novicesUse knowledge of experts to evaluate novices Is assessment only, doesn’t impact learningIs assessment only, doesn’t impact learning With clear expectations means you are “spoon-With clear expectations means you are “spoon-

feeding”feeding” Cannot be used in program assessmentCannot be used in program assessment


Rubrics as Assessment of Rubrics as Assessment of Program OutcomesProgram Outcomes

Learning Outcome

Direct Measure

Assessment Method

Students will be Students will be able to select able to select valid/ reliable valid/ reliable tests/measures tests/measures to assess to assess patient patient outcomesoutcomes

Assignment 2, Assignment 2, Neuromuscular Neuromuscular I: Given case, I: Given case, students select students select tests/measures tests/measures & provide & provide rationale based rationale based on patient on patient needs & psycho- needs & psycho- metric metric propertiesproperties

Rubric identifies Rubric identifies ability to ability to ascertain ascertain validity & validity & reliability of reliability of tests and tests and measures by measures by reading reading relevant relevant literature, and literature, and applying to applying to specific needs specific needs of a patient (see of a patient (see sample)sample)


Principles of Effective Principles of Effective GradingGrading11

1. Appreciate the complexity of grading2. Substitute judgment for objectivity3. Distribute time effectively 4. Be open to change5. Listen & observe6. Communicate & collaborate with students7. Integrate grading with other key processes8. Seize the teachable moment (feedback)9. Make student learning the primary goal10. Be a teacher FIRST, gatekeeper LAST11. Encourage learning-centered motivation12. Emphasize student involvement

11 Walvoord &Anderson (1998) Walvoord &Anderson (1998)


Types of Grading in Health Types of Grading in Health ProfessionsProfessions

Student Assessment Student Assessment MethodsMethods Multiple choice examinationMultiple choice examination Short answer questions on testsShort answer questions on tests Essay questionsEssay questions Annotated bibliographiesAnnotated bibliographies Literature reviewsLiterature reviews Case StudiesCase Studies Oral ExaminationsOral Examinations Practical/Performance Practical/Performance

ExaminationsExaminations Clinical AssessmentsClinical Assessments JournalsJournals PortfoliosPortfolios Lab ConclusionsLab Conclusions

2LC Jacobs, CI Chase (1992). Developing and using tests effectively. Jossey-Bass: San Francisco

Student-Constructed Responses


Methods of Grading Student-Methods of Grading Student-Constructed ResponsesConstructed Responses

Method Advantage DisadvantageNorm Referenced: : categorize work categorize work into A, B, C, D into A, B, C, D

Little work up Little work up front, recognizes front, recognizes faculty as faculty as ‘expert’, flexible‘expert’, flexible

-Potential for -Potential for biasbias

-Less opportunity -Less opportunity for learning, for learning, vaguevague

Checklist: list : list of criteria to of criteria to include include (introduction, (introduction, research research question…)question…)

Assignment Assignment directions match directions match checklist, not checklist, not difficult to difficult to prepareprepare

-Usually lacks -Usually lacks descriptionsdescriptions

-Lists + traits or -Lists + traits or behaviors, no behaviors, no negativenegative

Grading Rubric: criterion referenced, describe performance expectations & weighting

Explicit Explicit expectations, expectations, better feedback, better feedback, greater inter-greater inter-rater reliability, rater reliability, links to links to performanceperformance

-Takes time to -Takes time to construct, may construct, may need to modify need to modify after 1after 1stst run run


What are Assessment What are Assessment Rubrics?Rubrics?

Small Group DiscussionSmall Group Discussion In small group, look at the sample In small group, look at the sample

rubric packet. Then, answer the rubric packet. Then, answer the following:following:

Discussion QuestionsDiscussion Questions Which rubrics seem the MOST clear to Which rubrics seem the MOST clear to

you? Why?you? Why? Which rubrics are not clear? Why?Which rubrics are not clear? Why? What was common among the rubrics you What was common among the rubrics you

viewed as clear?viewed as clear? Based on samples, how do you define Based on samples, how do you define

assessment rubrics?assessment rubrics?


What is a Grading What is a Grading Rubric?Rubric?

Method of “articulating expectations for an Method of “articulating expectations for an assignment by listing the criteria, or assignment by listing the criteria, or what what countscounts, & describing , & describing levels of qualitylevels of quality from from excellent to poor”excellent to poor”33

Type of assessment that specifies Type of assessment that specifies gradations of gradations of qualityquality from excellent to poor from excellent to poor44

A A criterion-referencedcriterion-referenced method of grading using method of grading using highly specific grading criteria that are linked to highly specific grading criteria that are linked to objectivesobjectives3 HG Andrade, Y Du (2005). Students perspectives on rubric-referenced assessment. Practical Assessment, Research & Evaluation, Vol 10 (3).

4 HG Andrade ((2005). Teaching with rubrics: The good, the bad, and the ugly. Coll Teaching, 53 (1)


What is a Grading What is a Grading Rubric?Rubric?

PerformaPerformance Level nce Level 33



DimensioDimension 1n 1

3 pts: 3 pts: describedescribe

2 pts2 pts 1 pt1 pt


4 pts4 pts 2.5 pts2.5 pts 1 pt1 pt


3 pts3 pts 2 pts2 pts 1 pt1 pt


2 pts2 pts 1 pts1 pts 0 pts0 pts

Gradation: excellent-poorCategories

important to the teacher/class

Weighted PointsDimensions also called criteria


Criteria 1Criteria 1 Criteria 2Criteria 2 Criteria 3Criteria 3

Level 1Level 1 3 pts: 3 pts: describedescribe

2 pts2 pts 1 pt1 pt

Level 2Level 2 4 pts4 pts 2.5 pts2.5 pts 1 pt1 pt

Level 3Level 3 3 pts3 pts 2 pts2 pts 1 pt1 pt

Level 4Level 4 2 pts2 pts 1 pts1 pts 0 pts0 pts


How do Students Perceive How do Students Perceive Rubrics?Rubrics?

Andrad & Yu, 2005Andrad & Yu, 2005 Investigated how students use grading rubricsInvestigated how students use grading rubrics

Focus group / qualitative designFocus group / qualitative design Students reported that they used rubrics toStudents reported that they used rubrics to

Help them determine faculty expectationsHelp them determine faculty expectations To plan an approach to assignmentTo plan an approach to assignment Check/revise work before handing inCheck/revise work before handing in Help reflect on their learning-see Help reflect on their learning-see

strengths/weaknesses clearlystrengths/weaknesses clearly Perceived results of rubric usePerceived results of rubric use

Better, fairer gradesBetter, fairer grades Improvements in quality across classesImprovements in quality across classes Less anxietyLess anxiety


Using Primary Trait Using Primary Trait AnalysisAnalysis1,41,4

To Develop a Grading To Develop a Grading RubricRubricRationaleRationale

Analyze traits / characteristics of Analyze traits / characteristics of student learning and then clearly student learning and then clearly articulate them, leading to:articulate them, leading to: Assignment specificAssignment specific Explicit criteriaExplicit criteria Adds objectivity & consistency to holistic Adds objectivity & consistency to holistic

scoringscoring Lets students know in ADVANCE how will be Lets students know in ADVANCE how will be

gradedgraded

1 1 Walvoord &Anderson (1998)Walvoord &Anderson (1998)

44R Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell R Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell (Eds), Evaluating Writing: Describing, measuring, judging. Urbana IL: (Eds), Evaluating Writing: Describing, measuring, judging. Urbana IL: National Council of Teachers National Council of Teachers


Constructing a Primary Trait Constructing a Primary Trait Analysis ScaleAnalysis Scale11

1. Choose assignment/test that matches course objective

2. Identify all traits that will count for scoring in assessment

1. Body Language vs Thoroughness vs Accuracy

3. Build scale for scoring performance, gradations of skill

1. Scale usually ranges from 2-5 points

2. Include what should be demonstrated and what should be avoided

3. Build a range that discriminates A from A- from B+

4. Run scale by colleague, graduate, teaching assistant

5. Weight items: content > spelling; accuracy > efficiency, etc

4. Evaluate performance against criteria1. Try scale with sample & revise as needed


Example: Designing a Example: Designing a RubricRubric

Select assignment appropriate for PTASelect assignment appropriate for PTA Taking a Patient History (clinical (clinical

performance)performance) Identify ALL traits you look for in Identify ALL traits you look for in

patient historypatient history Body language & eye contact (Body language & eye contact (X-1X-1)) Thoroughness (Thoroughness (X-2X-2)) Data gathering (Data gathering (X 1.5X 1.5))

WWeighteight them each trait them each trait


Building a Scale: Building a Scale: Body Body LanguageLanguage

Level 3-ExcellentLevel 3-Excellent Consistently had good eye contact with patientConsistently had good eye contact with patient Communicated at patient’s eye-levelCommunicated at patient’s eye-level Maintained an open posture, leaning toward patientMaintained an open posture, leaning toward patient Facial expression interested, non-judgmentalFacial expression interested, non-judgmental Confident and relaxed, not arrogant Confident and relaxed, not arrogant

Level 2-GoodLevel 2-Good At least 4 of the above behaviors notedAt least 4 of the above behaviors noted OR, all 5 qualities demonstrated but not consistentlyOR, all 5 qualities demonstrated but not consistently

Level 1-FairLevel 1-Fair Inconsistent in several (2-3) areas, for example, confident Inconsistent in several (2-3) areas, for example, confident

through some portions of history but not others; or, eye through some portions of history but not others; or, eye contact missing at endcontact missing at end

Level 0-PoorLevel 0-Poor Arrogance, avoidance, lack of confidence detracted from Arrogance, avoidance, lack of confidence detracted from

relationshiprelationship

Gives students specific behaviors to demonstrate!Identifies behaviors to avoid!


Building a Scale: Building a Scale: ThoroughnessThoroughness

44: : Extremely ThoroughExtremely Thorough History includes all the following core areas: 1. all body History includes all the following core areas: 1. all body

systems; 2. history current illness/problem; 3. family history; 4. systems; 2. history current illness/problem; 3. family history; 4. medications; 5. educational background / learning style; 6. medications; 5. educational background / learning style; 6. lifestyle (diet, exercise, habits); 7. living/ work/social lifestyle (diet, exercise, habits); 7. living/ work/social environment; 8. assistive/adaptive devices (glasses, cane, etc); environment; 8. assistive/adaptive devices (glasses, cane, etc); 9. prior level of function 9. prior level of function

3: Thorough3: Thorough All items (1-9) included but may have missed some sub-All items (1-9) included but may have missed some sub-

categories (ie missed 2 body systems); or, only missed 1-2 of categories (ie missed 2 body systems); or, only missed 1-2 of the 9 core areas abovethe 9 core areas above

2: Somewhat Thorough2: Somewhat Thorough Missing 3-4 of the 9 core areas, or, missing only 1-2 areas but Missing 3-4 of the 9 core areas, or, missing only 1-2 areas but

was also superficial in review of body systemswas also superficial in review of body systems 1: Incomplete 1: Incomplete

Less than half of the 9 core areas were covered; incomplete Less than half of the 9 core areas were covered; incomplete datadata


Building a Scale: Building a Scale: Data Data GatheringGathering

Excellent-4Excellent-4 1. Uses open-ended, broad questions; 2. encourages patient 1. Uses open-ended, broad questions; 2. encourages patient

to tell story chronologically; 3. allows patient to talk without to tell story chronologically; 3. allows patient to talk without interrupting; 4. encourages by using phrases such as ‘tell interrupting; 4. encourages by using phrases such as ‘tell me more’ or ‘what else?’; 5. summarizes what was heard; 6. me more’ or ‘what else?’; 5. summarizes what was heard; 6. Approach was organizedApproach was organized

Good-3Good-3 Demonstrates at least 4 of 6 skills noted above throughout Demonstrates at least 4 of 6 skills noted above throughout

the history, or, used all 5 skills but inconsistently. the history, or, used all 5 skills but inconsistently. Fair-2Fair-2

Demonstrates 3 of the core probing skills, or, 3-4 skills used Demonstrates 3 of the core probing skills, or, 3-4 skills used however did so inconsistently; multiple ‘missed’ however did so inconsistently; multiple ‘missed’ opportunities to probe; disorganized at timesopportunities to probe; disorganized at times

Poor-1Poor-1 Did not consistently demonstrate probing skills, questions Did not consistently demonstrate probing skills, questions

were narrow and rarely open-ended, interrupted or cut were narrow and rarely open-ended, interrupted or cut patient off, rarely encouraged more information with patient off, rarely encouraged more information with phrases such as ‘tell me more’.phrases such as ‘tell me more’.


DIY: Build Your Own PTADIY: Build Your Own PTA

TraitsTraits

GradationsGradationsTrait 1: Trait 2: Trait 3:

Excellent

Good

Fair

Poor

Assignment or Test:____________________________________________________

Related Class Objectives: ________________________________________________


Step 4: Evaluate Step 4: Evaluate Performance Performance Using CriteriaUsing Criteria Test the ScaleTest the Scale

Use it with an past paper, videotape, case Use it with an past paper, videotape, case report, etc.report, etc.

What changes are needed in the rubric? What changes are needed in the rubric? Were the criteria easy to follow?Were the criteria easy to follow? Did the overall grade reflect performance?Did the overall grade reflect performance?

Are any changes needed in description Are any changes needed in description or in the point allotment?or in the point allotment? Did an A performance score better than a B?Did an A performance score better than a B?

Is it easy? Hard? Too cumbersome?Is it easy? Hard? Too cumbersome?


Applying the RubricApplying the Rubric Distribute ( or post) rubric to students in advanceDistribute ( or post) rubric to students in advance

Have conversations about expectationsHave conversations about expectations Ask students to attach rubric to assignmentAsk students to attach rubric to assignment

Helps students pay attention!Helps students pay attention! Use rubric to gradeUse rubric to grade

Match written comments to phrases in rubricMatch written comments to phrases in rubric Revise after useRevise after use

Make changes soon after grading for next timeMake changes soon after grading for next time Answer Curriculum QuestionsAnswer Curriculum Questions

Did the students learn? To what degree were objectives Did the students learn? To what degree were objectives met?met?

If not, was it the teaching? The assignment? If not, was it the teaching? The assignment? Background skills?Background skills?

What, if any, changes should be made in the class or What, if any, changes should be made in the class or curriculum? Rubric?curriculum? Rubric?


Advantages of Grading Advantages of Grading RubricsRubrics11

Save Save timetime in grading process in grading process Makes process of grading Makes process of grading reliable/fairreliable/fair Clarifies Clarifies expectationsexpectations for students for students Reinforces key Reinforces key conceptsconcepts - help faculty relate to - help faculty relate to

objectivesobjectives Students are Students are participantsparticipants as expectations are known as expectations are known Student Student peer reviewpeer review Works well if Works well if team-teachingteam-teaching Share Share across coursesacross courses or over curriculum or over curriculum Basis for departmental/Basis for departmental/programprogram assessmentassessment Which of these advantages pertain to your teaching

in the health professions?

1 Walvoord & Anderson, 19981 Walvoord & Anderson, 1998


CAUTION:CAUTION: Common Rubric Common Rubric PitfallsPitfalls

Rubric does not correspond with class or program Rubric does not correspond with class or program outcomesoutcomes Example: Entire rubric focused on writing quality not Example: Entire rubric focused on writing quality not

contentcontent Scale does Scale does notnot have enough gradations or levels have enough gradations or levels

Not distinguishing the A’s from the B+’sNot distinguishing the A’s from the B+’s All traits are given equal weight regardless of All traits are given equal weight regardless of

complexitycomplexity Grammar = ContentGrammar = Content

Too broad, not enough content describedToo broad, not enough content described Words like ‘breadth’ and ‘depth’ used in lieu of specificsWords like ‘breadth’ and ‘depth’ used in lieu of specifics Students still not sure, other grading faculty still not clearStudents still not sure, other grading faculty still not clear

Too long/too complicatedToo long/too complicated Faculty + students get lost in the rubricFaculty + students get lost in the rubric


Classroom Assessment Classroom Assessment TechniqueTechnique

Write down ONE thing you will commit to Write down ONE thing you will commit to incorporate regarding the use of grading incorporate regarding the use of grading rubricsrubrics

Write down ONE unanswered questionWrite down ONE unanswered question

Contact: [email protected]: [email protected]

designing grading rubrics to assess student learning the diy (do it yourself) workshop mary t....

Documents

designing grading rubrics

weaknesses of grading

role of grading rubrics

student learning

grading challenge

student assessment tool

mary blackinton

student motivation