module 21 using rubrics to assess learning. module 22 our journey a road map student involvement =...

Module 2 1

Module 2

Using Rubrics to Assess Learning

Module 2 2

Our JourneyA Road Map

Student Involvement

= Rest Stop, time for reflection

Creating a Rubric

Reliability and Validity

Target-Method-Match

Module 2 3

Assessment

Large-Scale

Assessment

Classroom Assessmen

t

Module 2 4

Assessment

Large-Scale Assessmen

t

Classroom Assessment

CLASSROOM ASSESSMENT

Formative Assessment

(for Learning)

Summative Assessment

(of Learning)

Purpose

Performance Assessment

Selected Response

Personal Communicatio

nEssay

Type

Module 2 5

Assessment

Large-Scale Assessmen

t

Classroom Assessment

CLASSROOM ASSESSMENT

Formative Assessment

(for Learning)

Summative Assessment

(of Learning)

Purpose

Performance Assessment

Selected Response

Personal Communication

Essay

Type

Module 2 6

Do you hear this at your school?

Why did I get a B for this assignment?

You didn’t tell us that spelling counts.

Will this be on the test?

Why do we have to learn this?

My friend said my “A” in America History isn’t the same as his “A”.

Module 2 7

Outcomes• Teachers will review their

understanding of quality assessment and its critical role in instruction.

• Teachers will know how to identify and develop appropriate criteria for use in rubrics and checklists.

• Teachers will know a process to develop performance assessment rubrics that are valid and reliable and inform instruction.

Module 2 8

In tr o d u c tio n /L e a d e r s h ip P r o f e s s io n a l

D e v e lo p m e n t M o d u le s H a w a ii S ta n d a r d s Im p le m e n ta tio n P r o c e s s

S ix S t e p M o d e l Id e n t if y R e le v a n t

S t a n d a r d s D e t e r m in e A c c e p t a b le e v id e n c e

D e t e r m in e L e a r n in g E x p e r ie n c e s

T e a c h a n d C o lle c t E v id e n c e

A s s e s s s t u d e n t w o r k /in f o r m in s t r u c t io n

E v a lu a t e /ju d g e / c o m m u n ic a t e le a r n in g r e s u lt s

# 1 . S t a n d a r d s Im p le m e n t a t io n P r o c e s s

H C P S T o o lkit: C u r r ic u lu m

F r a m e w o r ks In s tr u c tio n a l G u id e

: T h e p r o c e s s 6 s te p s C y c lic a l N a tu r e B u ilt o n s y s te m o f

s ta n d a r d s

# 2 U s in g R u b r ic s t o A s s e s s L e a r n in g

H C P S T o o lkit: C u r r ic u lu m

F r a m e w o r k In s tr u c tio n a l G u id e

L e a r n in g d r iv e n b y

s ta n d a r d s

K n o w w h e n

le a r n in g h a s b e e n a c h ie v e d

K n o w w h a t le a r n in g lo o ks like

E v id e n c e a lig n e d to c u r r ic u lu m a n d in s tr u c tio n

U s e in fo r m a tio n fo r g o a l s e ttin g

# 3 C u r r ic u lu m M a p p in g

H C P S T o o lkit: S c o p e a n d

S e q u e n c e G L P I

C u r r ic u lu m m a p s a n d a s s e s s m e n t d a ta

S c h o o l a n d

d is tr ic t r e fo r m a p p r o a c h

C o lle c t d a ta

r e g a r d in g a c tu a l te a c h in g

# 4 . S t a n d a r d s - B a s e d G r a d in g a n d R e p o r t in g

H C P S T o o lkit R C F A Q s /W e b s ite P a r e n t B r o c h u r e

A s s e s s m e n t G r a d e s

B a s e d o n S ta n d a r d s

A lig n m e n t w ith

s ta n d a r d s b a s e d in s tr u c tio n a n d a s s e s s m e n t

G r a d in g

P r a c tic e s P u r p o s e fo r

G r a d in g Is s u e s /S p e

c ia l N e e d s

G u id e lin e s H I Im p le m e n ta tio n

Module 2 9

Assess TheAmerican Idol

After viewing the following performances, give the performer a letter grade (A-F).

Module 2 10

American Idol: Got Assessment?

3

1

0

10

0

2

3

0

0

0

13

0

5

6

17

0 5 10 15 20

William

Jasmine

Fantasia

ABCDF

Module 2 11

Standards Implementation Planning Model Adapted from WestEd’s Learning from Assessment

Module 2 12

Student Involvement

Module 2 13

Student-Involved Assessment

• Are partners in development• Describe expectations for success• See how close they are now• Create a clear path to success

Module 2 14

Student-Involved Communication

• Student-led parent/teacher conferences

• Greater sense of responsibility• Pride in accomplishment• Greater achievement

Module 2 15

Creating a Rubric

Module 2 16

4-Step Process for Setting Criteria with

Students• Step 1: Brainstorm

• Step 2: Sort & Categorize

• Step 3: Create a T-chart

• Step 4: Add, Revise, & Refine

Module 2 17

Step 1: Brainstorm

• Pose a question

• Record all ideas on the chart paper

• Add in your own ideas to make sure

that all the important features are

included and your goals are met

Module 2 18

Step 2: Sort & Categorize

• Have students look at the list and see if any ideas fit together. Ask “Do you see any similar items?”

• Circle or mark the similar items using colored pens.

• Decide on headings using big ideas.

Module 2 19

Step 3: Create a T-Chart

• Draw a T-Chart• Transfer the items

from Step 2.• Put the big ideas or

criteria on the left.• Put the details or

specifics on the right.

Criteria

American Idol

Details

Module 2 20

Now that you have the criteria, what do you do with it??

Rubrics

Module 2 21

What is a Rubric?

• An assessment that evaluates student performance

• Uses specific criteria for assessment

• Defines levels of performance for each criterion

Module 2 22

Why Use Rubrics

• Assessment is more objective and consistent.

• The target is clearer – shows what is expected and how work will be evaluated.

• It gives feedback on the effectiveness of the instruction.

• It provides benchmarks against which to measure and document progress.

Module 2 23

Common Features of Rubrics

• Measures a stated objective (performance, behavior, or quality)

• Uses a range to assess performance

• Contains specific performance characteristics arranged in levels indicating the degree to which a criterion has been met.

Module 2 24

Terms to use in measuring range/scoring

levels• Needs Improvement … Satisfactory,

Good … Exemplary• Needs work … Good … Excellent• Beginning … Developing …

Accomplished… Exemplary

• Performance grid 5…3…1

Guidelines for Rubric Developmenthttp//edweb.sdsu.edu/triton/july/Rubrics/Rubric_Guidelines.html

Module 2 25

Step 4: Add, Revise, Refine

• Have students review the T-Chart.• Ask students, “Are there any new

items or criteria that we need to add?”

• Make changes as necessary.

Module 2 26

• What are the implications of involving students in assessment?

REST STOP

Module 2 27

Target

Method

Match

Module 2 28

Five Standards of Assessment Quality

Standard 1Clear Targets

Standard 2Clear and

Appropriate Users and UsesStandard 3

Appropriate Assessment Methods

Standard 4Sampling

Standard 5Potential Sources of Bias and Distortion

Module 2 29

TUMS Bias

T = Clear TargetsU = Appropriate Users &

UsesM = Method of

AssessmentS = SamplesBias = Minimize Bias

Module 2 30

The Various Possible Links Between AchievementTargets and Assessment Methods

TARGET TO

BE ASSESSED

ASSESSMENT METHOD

SELECTED RESPONSE

ESSAY PERFORMANCE

ASSESSMENT PERSONAL

COMMUNICATION

KNOWLEDGE MASTERY

Multiple choice, true/false, matching, and fill-in can sample mastery of elements of knowledge

Essay exercises can tap understanding of relationships among elements of knowledge

Not a good choice for this target--Three other options preferred

Can ask questions, evaluate answers and infer mastery--but a time-consuming option

REASONING

PROFICIENCY

Can assess understanding of basic patterns of reasoning

Written descriptions of complex problem solutions can provide a window into reasoning proficiency

Can watch students solve some problems and infer about reasoning proficiency

Can ask student to Òthink aloudÓ or can ask follow up questions to probe reasoning

SKILLS

Can assess mastery of the knowledge prerequisites to skillful performance--but cannot rely on these to tap the skill itself

Can observe and evaluate skills as they are being performed

Strong match when skill is oral communication proficiency; also can assess mastery of knowledge prerequisite to skillful performance

ABILITY TO CREATE

PRODUCTS

Can assess mastery of knowledge prerequisite to the ability to create quality products--but cannot use these to assess the quality of products themselves

A strong match can assess; (a) proficiency in carrying out steps in product development, and (b) attributes of the product itself

Can probe procedural knowledge and knowledge of attributes of quality products--but not product quality

DISPOSITIONS

Selected response questionnaire items can tap student feelings

Open-ended questionnaire items can probe dispositions

Can infer dispositions from behavior and products

Can talk with students about their feelings

Links Between Achievement Targets and Assessment Methods

Reproduced with permission from Rick Stiggins

Module 2 31

WHAT DO WE WANT TO ASSESS? — ACHIEVEMENT

TARGETS• Knowledge and

understanding• Reasoning• Performance

Skills• Products• Dispositions

Module 2 32

GENERAL LEARNER OUTCOMES

1. Self-Directed Learner2. Community Contributor3. Complex Thinker4. Quality Producer5. Effective Communicator6. Effective and Ethical User of

Technology

Module 2 33

WHAT ARE WAYS OF ASSESSING? —

ASSESSMENT METHODS• Selected response

• Essay

• Performance assessment

• Personal communication

Module 2 34

Selected Response Example

Benchmark: Explain the phases of the moon and eclipses.

A lunar eclipse occurs when:A. The moon passes between the earth and the sun.B. The earth passes between the moon and the sun.C. The sun passes between the earth and the moon.D. Every month when there is a “new moon”.E. Both answers A and D.

Module 2 35

Short ResponsesBenchmark: Describe and explain the effects of multiple forces acting on an object.Prompt: There are at least two forces that affect the speed of your car, forward motion and resistance to forward motion.

– Explain how these forces affect the performance of your solar car and give an example to support your answer.

Module 2 36

Personal Communications

Type of Target AchievementDispositions

Assessment MethodPersonal CommunicationCan talk with students about their feelings.

Fine Arts:Integrated Stds, Grade 1 Performance Indicator: Discusses personal feelings about a variety of works.

Make an animal sculpture with clay starting with a pinch pot. Tell how you feel about the creature that you made.

Module 2 37

Task+ Criteria

= Performance Assessment

Module 2 38

Task• Any activity, project,

prompt, problem or assignment that is given to the student to do.

• The quality of the student response to the task is what is assessed using the criteria.

Module 2 39

Criteria for SelectingPerformance Tasks

• Measure several standards or outcomes

• Involve critical thinking skills• Use instructional time wisely• Focus on credible real world tasks• Minimize bias and is fair• Is feasible• Is clearly defined-- Adapted from Herman, Aschbacher and Winters (1992)

Module 2 40

Activity1. Partner with someone and identify a grade

level.

2. Using the Curriculum Framework at your table,

identify standard(s) and the grade level

performance indicators (GLPI).

3. Based on the GLPI, identify the most

appropriate assessment target and method.

4. Develop performance task.

Module 2 41

What are your ah-has and questions on Target-Method-Match?

REST STOP

Module 2 42

Five Standards of Assessment Quality

Standard 1Clear Targets

Standard 2Clear and

Appropriate Users and UsesStandard 3

Appropriate Assessment Methods

Standard 4Sampling

Standard 5Potential Sources of Bias and Distortion

Module 2 43

Know your students

Observations

Communications Products

Module 2 44

Triangulation of Evidence: Assessment Data from

Many Sources

Watching Students

Interviews, Writings,Discussions

Student Work,Evidence

Module 2 45

Create Content Area Rubric Instructions

• In your groups, choose a sample performance assessment task in an Instructional Guide or use the task created in the prior activity.

• Create a content area rubric using the 4 Steps process for a performance assessment task.

Module 2 46

Decisions About Performance Criteria

Performance Criteria

Task Specific Generalized

Holistic (on overall

Score)

Analytical Trait -

several dimensions scored separately

Quantity or Quality

...Amount of Detail

Number of Score Points

Module 2 47

General or Task Specific?

• GENERALIZED or Generic - can be used across similar performances.

• TASK-SPECIFIC- each one can only be used for a single task.

Module 2 48

Holistic or Analytical?

HOLISTIC•gives a single score or rating for an entire product or performance based on an overall impression of a student’s work.

ANALYTICAL TRAIT

• divides a product or performance into essential traits or dimensions so that they can be judged separately.

Module 2 49

Holistic or Analytical Trait

HolisticUse :• Quick snapshot of overall

status• When speed of scoring is

important• Simple products or

performances

Disadvantages:• 2 students can get same

score for different reasons

• Can’t identify strengths & weaknesses

• Not useful for students

AnalyticalUse• Planning instruction - show

relative strengths & weaknesses

• Details to teach student quality

• Detailed feedback• Precision more important

that speed:Disadvantages:• Scoring is slower• Takes longer to learn

Module 2 50

Options for Selecting Rubrics

• Create your own - build from scratch• Adopt - use an existing rubric• Adapt - Modify or combine existing

rubrics– Reword parts– Drop or change one more scales– Omit criteria not relevant– “Mix” and Match”– Divide a holistic rubric into several scales– Add a “no response” category– Modify for different grade levels

Module 2 51

RELIABILITY AND VALIDITY

Reliability and

Validity

Module 2 52

Five Standards of Assessment Quality

Standard 1Clear Targets

Standard 2Clear and

Appropriate Users and UsesStandard 3

Appropriate Assessment Methods

Standard 4Sampling

Standard 5Potential Sources of Bias and Distortion

Module 2 53

Module 2 54

Board Policy 4501

Grades recorded by teachers must meet the dual criteria of validity and reliability.

•Validity: grades have been based on HCPS.

•Reliability: – Sufficient number of grade entries to allow

students ample opportunities over a grading period.

– Records are maintained accurately, legibly and support the grades given.

Module 2 55

ACTIVITY—Assessing student work with a rubric or checklist

1. Individually read and score student work samples one, two, and three using the rubric.

2. Share the scores in your group.3. Discuss the criteria on the rubric to develop

common understanding.4. Individually read and score student work

samples four to seven using the rubric.5. Share the scores in your group.

Module 2 56

Reflect on the following:

What do your teachers need to know to use rubrics effectively?

REST STOP

Module 2 57

Validity

A good rubric must possess “validity.”Definition: • the extent to which what is being

measured by an instrument is actually what is intended. Are the test and rubric actually measuring the desired performance-outcomes?

Module 2 58

Reliability

A good rubric must possess “reliability.”

Definition: • the extent to which the measuring

instrument yields responses that are consistent across time and between different scorers.

Module 2 59

Reliability• Factors that have nothing to do with the

purpose of the assessment can impact scores a student can get. Examples:– Rater fatigue– Rater’s mood– Failing students may get more critical look

than “good” students

• Well-designed rubrics respond to this concern by establishing a description of the scoring criteria in advance. Scorer should revisit the established criteria to ensure consistency.

Module 2 60

Reliability (cont.)

• Clarity of the rubric– Are the scoring categories well-defined?– Are the differences between scoring

categories clear?– Would two independent raters arrive at the

same score for a given response?

• Use anchor papers.• Whenever possible, share scoring

rubrics with students in advance.

Module 2 61

Process for Training and Managing Scorers

1. Duplicate a sampling of all tests and have all scorers evaluate and score each test.

2. Ask all scorers to review each other’s scoring.

3. Ask them to discuss discrepancies.4. Arrive at consensus on interpretation

and application of rubric criteria.5. Ask scorers to jointly-rescore tests.6. Periodically review each other’s tests.

Module 2 62

A Final Caution

• Establishing reliability is a prerequisite for validity.

• Although a valid assessment is by necessity reliable, the contrary is not true (e.g. you may have a reliable test that focuses too much on an element that is not relevant to the purpose of the assessment).

• On a reliable test, a student should get the same score regardless of when the test was taken, when it was scored and who scored it.

Module 2 63

What concerns or issues regarding reliability and validity need to be considered when developing and using rubrics?

REST STOP

Module 2 64

Rubric Development: Step 4

REVISE• Look at your American Idol rubric.• Consider these questions: - “Are there any new criteria we need to add?” - “Is there anything we’ve listed that someone doesn’t understand?” - “Have we included any criteria that are not

significant?”

• Add, Revise, Refine your rubric.

Module 2 65

Activity In your groups:• Select a song that can be performed

in 1 minute. • Practice the song so that you meet all

the performance criteria in the American Idol rubric.

• Perform the song to the large group.• Audience will assess the performance

using the American Idol rubric.

Module 2 66

Strategies for Using Rubrics

as Instructional Tools1. Understandable vision of the learning

target2. Use Models3. Descriptive Feedback4. Student Self-Assessment5. Focused on One Aspect6. Focused Revision7. Student Self-Reflection and

Communication About Learning

Module 2 67

Debrief• When is it appropriate to use rubrics?• How can I use Target-Method-Match to

guide my classroom assessment?• What are some strategies for involving

students in assessment?

Module 2 68

Part V: Planning Next Steps

Complex/District Planning Time…