tl1080presentation.pdf

8/14/2019 TL1080Presentation.pdf

1/59

1


2/59

CREATING RUBRICS FOR ASSESSMENT

2

TL1080


3/59

Using Rubrics for Student Assessment

Using

Rubrics forStudent

Assessment

3


4/59

Workshop Goals

4

Considering the variousrubric options

Creating a CustomizedRubric

Evaluating A Rubric foreffective assessment

Calibrating rubric scoring forreliable assessment data


5/59

Rubric Overview

Rubrics provide the criteria for classifying products or behaviorsinto categories that vary along a continuum.

They can be used to classify virtually any product or behavior, such asessays, research reports, portfolios, works of art, recitals, oralpresentations, performances, and group activities.

Judgments can be self-assessments by students; or judgments can bemade by others, such as faculty, other students, supervisors, orexternal reviewers.

Rubrics can be used to provide formative feedback to students, tograde students, and/or to assess programs.

There are two major types of scoring rubrics:

Holistic scoring: one global, holistic score for a product or behavior Analytic rubrics: separate, holistic scoring of specified characteristics

of a product or behavior.

5


6/59

Rubric Options

Considering the various rubric options

Nature of the rubric

Assessment of student

Knowledge

Performance

Assigning grades

Feedback for students

Assessment of curriculum Feedback for faculty

6


7/59

Rubric Options

Considering the various rubric options (continued):

Nature of the rubric

Aligned with the course syllabus and student learning outcomes

Designed to assess what is being taught

User friendly Can faculty scoring be calibrated to achieve reliable data?

Can students benefit from the rubrics feedback?

Holistic

Summative

Analytic Formative feedback

Quantitative

Qualitative

7


8/59

Rubric Options

What do you want to assess with your rubric?

Student

Knowledge: recall facts, terms, formulas

Skills: ability to use acquired knowledge

Performance: ability to produce a performance based

product

Student Learning Outcomes

Learning Objectives

8


9/59

Creating Rubrics

What do you want to assess with your rubric?

Student

Knowledge: recall facts, terms, formulas rather than a

rubric, a scantron test may be more suitable

Performance: ability to produce a

performance based product

An analytic rubric can generate a "profile" ofspecific student strengths and weaknesses

9


10/59

Creating Rubrics

Assigning Grades

Effective assessment and feedback for students

What are the outcomes and supporting objectives you want to

assess?

Have the students been given the opportunity to practice

these outcomes and objectives in class under the instructors

supervision?

Are these outcomes and objectives present in the course

syllabus?

10


11/59

Creating Rubrics

Assigning Grades

Effective assessment and feedback for students

Look for meaningful assessment categories, wording

and terminology that not only point outweaknesses and deficiencies, but also provide theinformation the student can use to understandwhy they received the grade they did as well as toimprove their performance in the future. Avoidincluding in your rubric any outcomes andobjectives that have not been included in thesyllabus or taught in the class.

11


12/59

Creating Rubrics

Assessing course curriculum

Effective assessment and feedback for faculty

Is the rubric instructionally aligned with the course

curriculum?

Does the rubric adequately address the student learning

outcomes and supporting objectives assigned to that course?

Does the rubric provide enough information to allow faculty

to pinpoint problems and weaknesses in the curriculum that

will need to be adjusted in the future?

12


13/59

Creating Rubrics

Assessing course curriculum

Effective assessment and feedback for faculty

For curriculum purposes, a rubric should assess a

courses student learning outcomes and the supportinglearning objectives, preferably one student learning

outcome and its objectives per rubric instrument.

A holistic rubric would be given first consideration as

they are less time consuming and not designed to

provide detailed feedback to students.

13


14/59

Creating Rubrics

Course curriculum

Syllabus and student learning outcomes are aligned

with actual classroom student performance products.

For rubric produced data to be reliable, the syllabus and its

student learning outcomes should be aligned with what is

actually being taught in the classroom.

Are full time faculty and adjuncts teaching the same material

and using the same assignments?

Are the textbooks and syllabi aligned with the curriculum?

14


15/59

Course curriculum alignment matrix

I = Introduced, D = Developed & Practiced with Feedback,

M = Demonstrated at the Mastery Level Appropriate for Graduation

15

Course Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5

1105 I, D I

1305 I D

1307 D D

1405 D D

1407 D

1411 D M

1412 D M

1413

1414 M DM M


16/59

Rubric Options

Course curriculum

Syllabus and student learning outcomes are aligned

with actual student performance

If there are too many variables between thecurriculum, the textbooks, the syllabi, the full time

faculty and the adjuncts; rubric produced data will be

unreliable.

It is critical to ensure there are alignment of course and

teaching variables, as well as calibration of grading and

scoring.

16


17/59

Rubric Options

Course curriculum

User friendly

Can faculty scoring be calibrated to achieve reliable data?

Is a wide variations in grading unusual?

Can students benefit from the rubrics feedback?

Should the learning process continue throughout the

testing or assessing process?

17


18/59

Rubric Options

Course curriculum

User friendly

Can faculty scoring be calibrated to achieve reliable data?

Faculty and adjuncts can be quickly trained tograde or score student work with standardized

results Can students benefit from the rubrics feedback?

Using clear and easily understandable wording inthe rubric provide students with helpful feedback

that will encourage improved performance

18


19/59

Creating Rubrics

Course curriculum

Holistic

Summative

The focus of a score reported using a holisticrubric is on the overall quality, proficiency, or

understanding of the specific content and skills.

Holistic rubrics can result in a less time consuming

scoring process than use of analytic rubrics.

19


20/59

Creating Rubrics

Course curriculum

Analytic

Student feedback

Quantitative

Qualitative

An analytic rubric articulates levels ofperformance for each criterion so instructor canassess student performance on each. Provides

detailed feedback for the students on theirstrengths and weaknesses. Most important, itcontinues the learning process.

20


21/59

Creating Rubrics

Course curriculum Analytic

Student feedback

Quantitative

Qualitative

The various levels of student performance can be defined usingeither quantitative (i.e., numerical) or qualitative (i.e.,descriptive) labels. In some instances, both quantitative andqualitative labels can be utilized. If a rubric contains four levelsof proficiency or understanding on a continuum, quantitativelabels would typically range from "1" to "4." When using

qualitative labels, there is much more flexibility, and can bemore creative. A common type of qualitative scale might includethese labels: master, expert, apprentice, and novice. Nearly anytype of qualitative scale will suffice, provided it "fits" with thetask.

21


22/59

Creating Rubrics

Rubrics support data driven decision making

Data-driven decision making (DDDM):

uses data on function, quantity and quality of inputs

Examines how students learn to suggest educationalsolutions

Based on the assumption that scientific methods can

effectively evaluate educational programs, and

instructional methods.

Rubrics provide significant data to support

effective DDDM.

22


23/59

Using Rubrics for Student Assessment

Break

23


24/59

Creating Rubrics

Creating a customized rubric

Step 1: Re-examine the learning objectives to be addressedby the task. This allows you to match your scoring guide withyour objectives and actual instruction.

Step 2: Identify specific observable attributes that you wantto see (as well as those you dont want to see) your studentsdemonstrate in their product, process, or performance.Specify the characteristics, skills, or behaviors that you will belooking for, as well as common mistakes you do not want tosee.

Step 3: Brainstorm characteristics that describe eachattribute. Identify ways to describe above average, average,and below average performance for each observable attributeidentified in Step 2.

24


25/59

Creating Rubrics


Step 4a: For holistic rubrics, write thorough narrativedescriptions for excellent work and poor workincorporating each attribute into the description.Describe the highest and lowest levels of performancecombining the descriptors for all attributes.

Step 4b: For analytic rubrics, write thorough narrativedescriptions for excellent work and poor work for each

individual attribute. Describe the highest and lowestlevels of performance using the descriptors for eachattribute separately.

25


26/59

Creating Rubrics


Step 5a: For holistic rubrics, complete the rubricby describing other levels on the continuum thatranges from excellent to poor work for the

collective attributes. Write descriptions for allintermediate levels of performance.

Step 5b: For analytic rubrics, complete the rubricby describing other levels on the continuum that

ranges from excellent to poor work for eachattribute. Write descriptions for all intermediatelevels of performance for each attribute separately.

26


27/59

Creating Rubrics


Step 6: Collect samples of student workthat exemplify each level. These will help

you score in the future by serving asbenchmarks.

Step 7: Revise the rubric, as necessary. Beprepared to reflect on the effectiveness ofthe rubric and revise it prior to its nextimplementation.

27


28/59

Creating RubricsTable 2: Template for analytic rubrics

Criteria #1 Beginning

1

Developing

2

Accomplished

3

Exemplary

4

Score

Criteria #2 beginning level ofperformance

movement toward

mastery level of

performance

achievement of

mastery level of

performance

highest level of

performance


movement toward

mastery level of

performance

achievement of

mastery level of

performance

highest level of

performance


movement toward

mastery level of

performance

achievement of

mastery level of

performance

highest level of

performance


movement toward

mastery level of

performance

achievement of

mastery level of

performance

highest level of

performance

28


29/59

Creating Rubrics

Table 1:

Template for Holistic Rubrics

Score Description

5Demonstrates complete understanding of the problem.

All requirements of task are included in response.

4Demonstrates considerable understanding of the

problem. All requirements of task are included.

3Demonstrates partial understanding of the problem.

Most requirements of task are included.

2Demonstrates little understanding of the problem. Manyrequirements of task are missing.

1 Demonstrates no understanding of the problem.

29


30/59

Creating Rubrics

Assigning scores to the rubric

If a rubric contains four levels of proficiency or understanding on a continuum,quantitative labels would typically range from "1" to "4." When using qualitativelabels, teachers have much more flexibility, and can be more creative. A commontype of qualitative scale might include the following labels: master, expert,apprentice, and novice. Nearly any type of qualitative scale will suffice, provided it"fits" with the task.

One potentially frustrating aspect of scoring student work with rubrics is the issueof somehow converting them to "grades." It is not a good idea to think of rubricsin terms of percentages (Trice, 2000). For example, if a rubric has six levels (or"points"), a score of 3 should not be equated to 50% (an "F" in most letter gradingsystems). The process of converting rubric scores to grades or categories is more aprocess of logic than it is a mathematical one. Trice (2000) suggests that in a rubricscoring system, there are typically more scores at the average and above average

categories (i.e., equating to grades of "C" or better) than there are below averagecategories. For instance, if a rubric consisted of nine score categories, theequivalent grades and categories might look like this:

30


31/59

Creating Rubrics

31

Table 3: Sample grades and categories

Rubric Score Grade Category

8 A+ Excellent

7 A Excellent

6 B+ Good

5 B Good

4 C+ Fair

3 C Fair

2 U Unsatisfactory1 U Unsatisfactory

0 Y Unsatisfactory


32/59

Hands on Practice

Create a holistic or analytical rubric for a course in your

discipline

Use blank rubric forms in your handbook

32


33/59

Rubrics and Assessment

Break

33


34/59

Evaluating A Rubric

Evaluating A Rubric for effective assessment

A Rubric for Rubrics: A Tool for Assessing theQuality and Use of Rubrics in Education

Downloaded January 22, 2010 from

http://webpages.charter.net/bbmullinix/Rubrics/A%20

Rubric%20for%20Rubrics.htm

Dr. Bonnie B. Mullinix Monmouth University

December 2003

34


35/59

Evaluating A RubricCriteria 1 Acceptable 3 Acceptable 3 Exemplary Exemplary

Clarity of criteria Criteria being

assessed are

unclear,

inappropriate

and/or

have significant

overlap

Criteria being

assessed can be

identified, but are

not clearly

differentiated or are

inappropriate

Criteria being

assessed are clear,

appropriate and

distinct

Each criteria is

distinct, clearly

delineated and fully

appropriate for the

assignment(s)

/course

Distinction

between

Levels

Little/no distinction

can be made

between levels of

achievement

Some distinction

between levels is

made, but is not

totally clear how

well

Distinction between

levels is apparent

Each level is distinct

and progresses in a

clear and logical

order

Distinction

betweenLevels

Rubric is not shared

with learners

Rubric is shared and

provides some ideaof the assignment/

expectations

Rubric is referenced

- used to introducean

assignment/guide

learners

Rubric serves as

primary referencepoint for discussion

and guidance for

assignments as well

as evaluation of

assignment(s),

35


36/59


Reliability of

Scoring

Cross-scoring

among faculty

and/or students

often results in

significant

differences

Cross-scoring by

faculty and/or

students

occasionally

produces

inconsistent

results

There is general

agreement

between

different scorers

when

using the rubric

(e.g. differs by

less than 5-10%

or less than

level)

Cross-scoring of

assignments

using rubric

results in

consistent

agreement

among scorers

Clarity of

Expectations/

Guidance to

Learners

Rubric is not

shared with

learners

Rubric is shared

and provides

some idea of theassignment/

expectations

Rubric is

referenced - used

to introduce anassignment/

guide learners

Rubric serves as

primary

reference pointfor discussion

and guidance for

assignments as

well as valuation

of assignment(s)

36


37/59


Support of

Metacognition

(Awareness of

Learning)

Rubric is not shared

with learners

Rubric is shared but

not discussed/

referenced with

respect to what is

being learned

through the

assignment(s)

/course

Rubric is shared and

identified

as a tool for helping

learners to

understand what

they are learning

through the

Assignment

/ in the course

Rubric is regularly

referenced

and used to help

learners identify

the skills and

knowledge they are

developing

throughout the

course/

assignment(s)

Engagement of

Learners in

Rubric

Development/Use *

Learners are not

engaged in

either development

or use of the

rubrics

Learners offered

the rubric and

may choose to use

it for self

assessment

Learners discuss the

design of the rubric

and offer Feedback

/ input and are

responsible for use

of rubrics in

peer and/or self-

evaluation

Faculty and learners

are jointly

responsible for

design of rubrics

and learners use

them in peer

and/or self-

evaluation

37


38/59

Calibrating a Rubric

Calibrating rubric scoring for reliable assessmentdata. The validity of your rubric and assessment depends in

part on the validation of the rubric scores.

Therefore calibration is an essential process. Calibration training for a group of instructors who can

then score rubrics and produce valid data is critical.

The process for calibration will be determined by each

discipline or program and may involve paired scoringor open table scoring. Rubric scores are determinedby consensus and part of the calibration expert's roleis to resolve discrepancies in scoring.

38


39/59

Calibrating Rubric Scoring

Each participant should have a copy of the task,samples of student work, extra rubrics, and scoringsheets.

Explain procedure (as below)

Ground rules: Only one person speaks at a time

Wait to be called on

No side conversations, please

Respect each others comments Designate a recorder to note any issues, record

discussions, and initial and final scores for each box.

39


40/59


Examine task for standards, instructor and studentdirections, prompt, and rubric.

Discuss and record.

Read first sample of student work only.

Score individually, marking rubric.

When determining final score, consider the preponderanceof evidence to determine where a students work falls.

Check the teacher directions page for any accommodationsthat might have been made for students with Letters of

Accommodation. When everyone has finished, ask for a show of hands for an

overall score of exceeds, meets, etc.

40


41/59


Work box by box, asking for an overall initial score. Then ask volunteers to support the evaluation, citing

evidence from the student work.

Generally, start with someone who is most supportive of

student work, work towards below categories. This discussion takes quite a bit of time, depending upon

the task and the sample of student work.

At the end of the discussion of each box, take a revote todetermine the final score for that box.

When all boxes have been discussed, ask for a finaloverall score.

Use same procedure for sample two.

41


42/59


OR (depending upon time and level of expertise)

pair score sample two and discuss as needed after an

overall vote.

Read Central scoring calibration notes forcomparison/validity purposes.

Pair score tasks, using score sheets.

Second scorer should not look at first evaluators

score until his/her own evaluation is completed.

42


43/59


At that point, if both scores agree, record the score

on the cover of the task. (Both scorers should also fill

out a score sheet).

If there is a discrepancy of a 3-4 or a 2-3, then adiscussion takes place to determine the final score.

If agreement is reached, one of the scorers changes

his/her score sheet to agree with the other score.

43


44/59

Glossary of Terms

Analytic Rubric: An analytic rubric articulates levelsof performance for each criterion so the teacher canassess student performance on each criterion.

Assessment:The systemic process of gathering and

analyzing information about student learning to ensurethe program and the students are achieving the desiredresults in relation to the student learning outcomes.

Assessment plan:a document which states the

procedures and processes used to gather and interpret

student learning outcome data.

44


45/59

Glossary of Terms

Authentic Task: An assignment given to studentsdesigned to assess their ability to apply standards-driven

knowledge and skills to real-world challenges. A task is

considered authentic when 1) students are asked to construct

their own responses rather than to select from ones

presented; and 2) the task replicates challenges faced in the

real world. Good performance on the task should

demonstrate, or partly demonstrate, successful completion of

one or more standards. The term task is often usedsynonymously with the term assessment in the field of

authentic assessment.

45


46/59

Glossary of Terms

Blooms Taxonomy: A classification of learning behaviors orcategories used as a way of determining learning progressionin a course or program. The revised taxonomy includes lowerlevel skills and high order thinking skills. Blooms Taxonomy isused to align Student Learning Outcomes and their

subsequent objectives. Calibration: Training faculty to score rubrics in a similar

fashion to ensure validity of scores and subsequent data.

Closing the Loop: Primarily regarded as the last step in theassessment process, closing the loop actually starts the

process over again if the data is analyzed and the desiredresults are not achieved. Closing the loop refers specifically tousing the data to improve student learning.

46


47/59

Glossary of Terms

Course Goals: Generally phrased, non measureablestatements about what is included and covered in a course.

Course Guide: a booklet and online resource that helps

students select subjects.

Curriculum Guide: a practical guide designed to aidteachers in planning and developing a teaching plan for

specific subject areas.

47


48/59

Glossary of Terms

Course Objectives: A subset of student learningoutcomes, course objectives are the specific teachingobjectives detailing course content and activities.

Criteria: Characteristics of good performance on aparticular task. For example, criteria for a persuasiveessay might include well organized, clearly stated,and sufficient support for arguments. (The singularof criteria is criterion.

Data-driven decision making: A process of makingdecisions about curriculum and instruction based onthe analysis of classroom data, rubric assessment,and standardized test data.

48


49/59

Glossary of Terms

Descriptors: Statements of expectedperformance at each level of performance fora particular criterion in a rubric - typically

found in analytic rubrics. See example andfurther discussion of descriptors.

Direct Assessment Method: The assessment isbased on an analysis of student behaviors or

products in which they demonstrate how wellthey have mastered learning outcomes.

49


50/59

Glossary of Terms

Indirect Assessment Method: The assessmentis based on an analysis of reportedperceptions about student mastery of learningoutcomes or the learning environment.

Instructional Alignment: the process ofensuring that Student Learning Outcomes andthe subsequent objectives use the samelearning behaviors or categories in BloomsTaxonomy.

50


51/59

Glossary of Terms

Program Assessment: An on-going, systemic processdesigned to evaluate and improve student learning byidentifying strengths and areas for improvement. The datafrom the evaluation is used to guide decision making for theprogram.

Reliability: The degree to which a measure yields consistentresults.

Rubric: the criteria for classifying products or behaviors intocategories which varies along a continuum. Rubrics are usedas a way of assessing a Student Learning Outcome. A scoring

scale used to evaluate student work. A rubric is composed ofat least two criteria by which student work is to be judged ona particular task and at least two levels of performance foreach criterion.

51


52/59

Glossary of Terms

Student Learning Outcomes: Student LearningOutcomes are defined as the specific knowledge, skills,or attitude students should be able to demonstrateeffectively at the end of a particular course or program.Student Learning Outcomes are measured and providestudents, faculty, and staff the ability to assess studentlearning and instruction. Each course should have four

to seven Student Learning Outcomes. Validity: The degree to which a certain inference from a

test is appropriate and meaningful.

52


53/59

Student Learning Cycle

Course Goal

Student Learning

Outcomes

CourseObjectives

SLO Rubric andAssessment

Program

Assessment

53


54/59

Blooms Taxonomy

First published in 1956, a classification of levelsof learning behaviors or categories.

The elements of the original Taxonomy includedKnowledge, Comprehension, Application,

Analysis, Synthesis, and Evaluation.

The revised Taxonomy now includes, in thesame order, Remembering, Understanding,

Applying, Analyzing, Evaluating, and Creating.

The Taxonomy is often used as a way ofdetermining the progression of learning or

intellectual skills in a program.

More advanced course work, for example, should relyon higher order skills while developmental or

elementary course work could focus on lower level

skills.

54


55/59

Blooms Pyramid

55

Evaluation

Level 6

Synthesis

Level 5

Analysis

Level 4

Application

Level 3

Comprehension

Level 2

Knowledge

Level 1

Bl ' T f C iti L l


56/59

Bloom's Taxonomy of Cognitive Levels

Verb Table

1 Knowledge 2 Comprehension 3 Application

list

name

identify

show

define

recognize

recall

state

summarize

explain

put into your own

words

interpret

describe

compare

paraphrase

differentiate

demonstrate

visualizefind more

information about

restate

solve

illustrate

calculate

use

interpret

relate

manipulate

apply

classify

modify

put intopractice

4 Analysis 5 Synthesis 6 Evaluation

analyze

organize

deduce

choose

contrast

compare

distinguish

design

hypothesize

support

schematize

write

report

discuss

plan

devise

compare

createconstruct

evaluate

choose

estimate

judge

defend

criticize

justify

56


57/59

Instructional Alignment

SLOs are the foundation for teaching

and learning district-wide.

Employ the Blooms Taxonomy ofCognitive Levels verb chart

Align Course Goal, Student LearningOutcomes, Course Objectives,

and Rubric Assessment

A level five SLO should be aligned withlevel five course objectives

as well as a level five rubric assessment

57


58/59

Resource Links

http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01

58
http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01http://www.hccs.edu/hccs/faculty-staff/student-learning-outcomes--01


59/59

THE CENTER for TEACHING & LEARNING EXCELLENCE

Online Resources: http://hccs.edu/tle
http://hccs.edu/tlehttp://hccs.edu/tle

tl1080presentation.pdf

Documents