carla rubrics

7/22/2019 Carla Rubrics

1/14

1

Process: Types of rubrics

Rubrics are generally categorized as genericor task-specific. As is so often the case in assessment,

the line between the two categories may blur so that rating instruments appear more or lessgeneric

or task-specific. Indeed, many task-based rubrics are adaptations of generic scales. It is also possible

to design hybrid rubrics that combine features of both types.

Generic rubrics can be applied to a number of different tasks. In language assessment, one

frequently finds generic rubrics used with assessment tasks within a modality (generally writing and

speaking) or mode (interpersonal and presentational). A truly generic rubric could be applied to any

task within the same modality or mode.

The dimensions in a generic rubric for second-language assessment often emphasize features of

language production, such as comprehensibility, accuracy, and vocabulary, without making reference

to specific content or task details. Generic rubrics are often derived from models of language

proficiency and/or second language acquisition.

Figure 1. Generic Rubric for Oral Presentations - Intermediate Level Learners

Exemplary4

Accomplished3

Developing2

Beginning1

Comprehen-sibility

Listeners

accustomed tothe speech oflearners are ableto understand allof thepresentation.

Listeners

accustomed tothe speech oflearners are ableto understandmost of thepresentation.

Listeners

accustomed tothe speech oflearners are ableto understand themain ideas andsome details ofthe presentation.

Listeners

accustomed tothe speech oflearners are ableto understandisolated bits ofthe presentation.

Text Type

Describes,narrates, and/orexpresses ownthoughts inparagraph-leveldiscourse.

Describes,narrates, and/orexpresses ownthoughts inconnected stringsof sentences.

Speaks in looselyconnectedsentences.

Speaks inunconnectedsentences andphrases.

LanguageControl

High degree ofaccuracy ingrammar andword choice in

connected,rehearsed, andoccasionallycomplexdiscourse. Littleor no interferencefrom firstlanguage.

Usually accurategrammar andword choice inconnected,

rehearseddiscourse.Occasionalinterference fromfirst language.

Frequent, butusually minor,grammar andword choice

errors inrehearsed,sentence-leveldiscourse.Significantinterference fromfirst language.

Comprehension isimpeded byfrequentgrammar and

word choiceerrors inrehearseddiscourse. Highdegree ofinterference fromfirst language.
http://www.umn.edu/


2/14

2

VocabularyUse

Uses a broad

range of familiarand new words,phrases, andidioms so that

expression ishighly varied andnon-repetitive.

Uses an adequate

range of familiarand new words,phrases, andidioms so that

expression isvaried and onlyoccasionallyrepetitive.

Uses familiar

words, phrases,and idioms, andrarely attemptsto go beyond

basic vocabulary.Speech isrepetitive andlacks variety.

Uses very basic

vocabulary andmemorizedphrases. Speechis limited and

highly repetitive.

CommunicationStrategies

Always maintainscommunication.Able tocircumlocute andself-correct whenneeded. Use ofmemory aidsenhancespresentation.

Very few breaksincommunication.Sometimes ableto circumlocuteand self-correct.Effective use ofmemory aids.

Frequent breaksincommunication.Rarely able tocircumlocute orself-correct. Useof memory aidssometimesdetracts frompresentation.

Generally unableto maintaincommunication.Overreliance onmemory aidsdetracts frompresentation.

Figure 1shows a sample generic, analytic rubric for oral presentations (presentational mode) for

Intermediate level learners adapted from theACTFL Performance Guidelines for K-12 Learners(K-12

Guidelines). This example reflects a focus on features of second-language production, but additional

dimensions might be included in order to measure such aspects of oral presentation as content

coverage, organization, connection with audience, elocution, use of graphics, and so on.

The terminology in Figure 1 is accessible to language-teaching professionals, but it may not provide

meaningful feedback to learners. Large-scale and external assessments for purposes such as

certification, placement, articulation, and program evaluation often use generic scales that contain a

high degree of professional language and require modification for classroom use.

A high school French teacher who commented on the rubric in Figure 1 indicated that, for classroom

use, he prefers a rubric with short descriptors that he can take in at a glance and that serve primarily

to refresh his memory of what performance is like at each step on the scale. Because he has only a

few seconds to evaluate each student, and because he wants to spend as little class time as possible

explaining the terms in the rubric to his students, prefers simple vocabulary that neither he nor his

students must ponder (J.-L. Roche, personal communication, October 22, 2002). Figure 2 presents an

adaptation of the oral presentation generic rubric for classroom use.

Figure 2. "Classroom-Friendly" Generic Rubric for Oral Presentations - IntermediateLevel Learners

Exemplary

4 Accomplished

3 Developing

2 Beginning

1

Comprehen-sibility

Listeners canunderstand all ofthe presentation.

Listeners canunderstand almost allof the presentation.

Listeners canunderstand themain ideas andsome details.

Listeners canunderstand somephrases orsentences.

Connected

Language

Speaks in

paragraphs todescribe, tell

about a sequenceof events, orexpress thoughts.

Speaks in sentences

to describe, tellabout a sequence of

events, or expressthoughts.

Sentences are

looselyconnected.

Phrases and

sentences areunconnected.

LanguageControl

Makes raregrammar orvocabulary errorsin preparedspeech.

Makes somegrammar orvocabulary errors inprepared speech.

Makes frequentgrammar orvocabularyerrors inpreparedspeech.

Makes so manyerrors that itappears speechwas notprepared.

Vocabulary

Use

Uses manyfamiliar and newwords, phrases,

and expressions.

Not repetitive.

Uses an adequaterange of familiar andnew words, phrases,

and expressions.

Occasionallyrepetitive.

Uses familiarand a few newwords, phrases,

and expressions.

Repetitive.

Uses very basicvocabulary andmemorized

phrases. Very

repetitive.
http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999http://www.carla.umn.edu/assessment/vac/references.html#actfl_1999


3/14

3

CommunicationStrategies

May glance atnotes. Nonoticeable pausesor hesitations.

May rely on notesseveral times. A fewnoticeable pauses orhesitations.

Relies on notesoften. Frequentnoticeablepauses orhesitations.

Unable to speakwithout readingnotes.

It is certainly most efficient to design or identify rubrics that can be used for multiple purposes, but

when weighing the use of generic versus task-based rubrics, efficiency is not the only important

criterion.Tedick (2002)writes: "While some rubrics are created in such a way as to be generic in

scope for use with any number of writing or speaking tasks, it is best to consider the task first and

make sure that the rubric represents a good fit with the task and your instructional objectives. Just as

a variety of task-types should be used in language classrooms, so should a variety of rubrics and

checklists be used for assessing performance on those tasks. (p. 37)" For learners who are new to

performance assessment and evaluation, Tedick recommends making students comfortable with the

process by first using generic rubrics and gradually introducing task-specific rubrics.

Task-specific rubrics are used with particular tasks, and their criteria and descriptors reflect specific

features of the elicited performance. Rubrics developed for a defined group of tasks within a modality

or mode, such as writing narratives, performing role-plays, or exchanging e-mail messages may

combine elements of language production with dimensions related to the content and language

function(s) of the lesson/task. For example, if an assessment task requires learners to use a series of

pictures to tell a story in the past about a visit to monuments in Paris, the scoring criteria would focus

on language competencies related to narration in past tense along with one or more dimensions

measuring content and cultural knowledge. A possible rubric for this task is shown in Figure 3.

Figure 3. Task-specific rubric for "A visit of Paris monuments"

4 3 2 1

Narration

Story is wellorganized andconnected;includes manydetails or

elaboration aboutall pictures in thetask.

Story is mostlyorganized andconnected;includes somedetails or

elaboration aboutmost pictures inthe task.

Story is sequenceof looselyconnected events;includes fewdetails or

elaboration aboutpictures in thetask.

Story isunconnected listof events;includes no detailsor elaborationabout pictures inthe task.

Use ofpast tense

Uses past tense at

all appropriatetimes; use isaccurate.

Uses past tensefrequently; use ismostly accurate.

Uses other tensessometimes where

past isappropriate; use isaccurate some ofthe time.

Makes fewattempts to usepast tense; use isfrequentlyinaccurate.

Culturalknowledge

Demonstratesextensive andcorrect knowledgeof current andhistoricalsignificance of allmonuments.

Demonstratesadequate andcorrect knowledgeof current andhistoricalsignificance ofmost monuments.

Demonstratespartial and usuallycorrect knowledgeof current andhistoricalsignificance ofsome monuments.

Demonstratesminimal or noknowledge ofcurrent andhistoricalsignificance ofmonuments.

Rubrics that combine features of generic and task-specific rubrics are very useful in classroom

assessment because they provide feedback to learners on broad dimensions of language production

along with their performance on the particular competencies and knowledge targeted by course

content and aligned assessments. When adapting the rubrics for other tasks, teachers may keep the

generic language production elements as they are and change one or two categories to focus on task

expectations. For example, one might add level-appropriate, generic dimensions such as pronunciation

or fluency to the task-specific categories of narration, use of past tense, and knowledge about

monuments of Paris to the rubric in Figure 3.
http://www.carla.umn.edu/assessment/vac/references.html#tedick_2002http://www.carla.umn.edu/assessment/vac/references.html#tedick_2002http://www.carla.umn.edu/assessment/vac/references.html#tedick_2002http://www.carla.umn.edu/assessment/vac/references.html#tedick_2002


4/14

4

Holistic, analytic, primary traitand multiple traitrubrics may be seen as different ways of

selecting and organizing rating criteria. These rubric types come from different contexts, and although

their particular uses and characteristics have converged in current practice, there are some general

guidelines for choosing among them. In addition, each type has advantages and disadvantages.

In practice, you will probably find considerable variability in how rubric types are identified. Holisticand analytic scales may be identified as generic or task-specific, or they may include rating criteria of

both types. Primary and multiple trait rubrics are essentially task-specific, but general language

production categories may be added to multiple trait rubrics.

Evaluation

Process

Types of rubrics: Holistic scales

In holistic evaluation, raters make judgments by forming an overall impression of a performance and

matching it to the best fit from among the descriptions on the scale. Each band on the scale describes

performance on several criteria (e.g., range of vocabulary + grammatical accuracy + fluency). Four or

six levels of performance are commonly found in holistic rubrics. Holistic scales may be either generic

or task-specific. Large-scale assessments are often evaluated holistically, but teachers find holistic

rubrics easy and efficient to use for classroom assessment as well.

Fig. 1a Holistic rubric for speaking tasks (generic)

Exceeds

expectations

No errors in expression; near-native pronunciation; use of structures beyond

expected proficiency; near-native use of appropriate cultural practices;

exceeded task requirements.

Meets

expectations

Almost all expression is correct; easily understood with infrequent errors in

pronunciation, structures, and vocabulary usage; almost all cultural practicesdemonstrated and appropriate; met task requirements.

Developing

Some errors in expression; comprehensible with noticeable errors in

pronunciation, structures, and/or vocabulary usage; some cultural practices

demonstrated and appropriate; met most task requirements.

Not there

yet

Little or no expression is correct; nearly or completely incomprehensible;cultural practices were inappropriate or not demonstrated at all; little success

in meeting task requirements.

Adapted from sample rubric in theNew Jersey World Languages Curriculum Framework, Appendix B(PDF document).

Figures 1a-d present two holistic rubrics for speaking tasks (both generic) and two holistic rubrics for

writing tasks (one generic, one task-specific). Click the icon at left to open a new window displaying

Figures 1a-d.

Advantages:

They are often written generically and can be used with many tasks.

They emphasize what learners can do, rather than what they cannot do.

They save time by minimizing the number of decisions raters must make. Trained raters tend to apply them consistently, resulting in more reliable measurement.

They are usually less detailed than analytic rubrics and may be more easily understood by

younger learners.

Disadvantages:

They do not provide specific feedback to test takers about the strengths and weaknesses of

their performance.

Performances may meet criteria in two or more categories, making it difficult to select the one

best description. (If this occurs frequently, the rubric may be poorly written.)

Criteria cannot be differentially weighted.
http://www.state.nj.us/njded/frameworks/worldlanguages/appendb.pdfhttp://www.state.nj.us/njded/frameworks/worldlanguages/appendb.pdfhttp://www.state.nj.us/njded/frameworks/worldlanguages/appendb.pdfhttp://www.state.nj.us/njded/frameworks/worldlanguages/appendb.pdf


5/14

5

Evaluation

Process

Types of rubrics: Analytic scales

Analytic scales are usually associated with generic rubrics and tend to focus on broad dimensions of

writing or speaking performance. These dimensions may be the same as those found in a generic,

holistic scale, but they are presented in separate categories and rated individually. Points may be

assigned for performance on each of the dimensions and a total score calculated.

Traditionally, analytic rubrics are associated with large-scale assessment of general dimensions of

language performance. However, analytic rubrics certainly can be created or adapted for use in

classroom settings and with particular tasks (e.g.,Taggart et al., 1998). These rubrics often combine

performance categories from a generic rubric with categories directly related to a task, such as

demonstrating understanding of specific lesson content (Moskal, 2000). In practice, the names

"analytic rubric" and "multiple trait rubric" may be used interchangeably.

Performance dimensions commonly found in analytic rubrics include:

Speaking & Writing

Content

Vocabulary

Accuracy/Grammar/

Language Use

Task fulfillment Appropriate use of

language

Creativity

Sentence

structure/Text type

Comprehensibility

Writing

Organization

Style

Mechanics

Coherence and

Cohesion

Speaking

Fluency

Pronunciation

Intonation

FromTedick,p. 35:

"One of the best known analytic rubrics used for writing assessment in the field of English as asecond language (ESL) was developed byHughey et al. (1983, p. 140). This rubric has fivecategoriescontent, organization, vocabulary, language use, and mechanics. Drawing heavilyupon characteristics of the Hughey et al. scale, Tedick and Klee developed an analytic rubric foruse in scoring essays written for an immersion quarter for undergraduates studying Spanish

(Klee, Tedick, & Cohen 1995)."

Fig. Fx. Analytic Writing Scale for the Spanish Foreign Language Immersion ProgramUniversity of Minnesota, Revised July, 1996

CONTENT 30 POINTS POSSIBLE

Score Range Criteria Comments

30 - 27

Excellent to Very Good.addresses allaspects of the prompt .provides good supportfor and development of all ideas with range ofdetail .substantive

26 - 22

Good to Average.prompt adequately

addressed .ideas not fully developed orsupported with detail, though main ideas areclear .less substance

21 - 17

Fair.prompt may not be fully addressed(writer may appear to skirt aspects of prompt).ideas not supported well, main ideas lackdetailed development .little substance

16 - 13Poor.doesnt adequately address prompt.little to no support or development of ideas.non-substantive
http://www.carla.umn.edu/assessment/VAC/references.html#taggart_1998http://www.carla.umn.edu/assessment/VAC/references.html#taggart_1998http://www.carla.umn.edu/assessment/VAC/references.html#taggart_1998http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/gloss_windows/references.html#POLIAhttp://www.carla.umn.edu/assessment/VAC/gloss_windows/references.html#POLIAhttp://www.carla.umn.edu/assessment/VAC/gloss_windows/references.html#POLIAhttp://www.carla.umn.edu/assessment/VAC/references.html#hughey_1983http://www.carla.umn.edu/assessment/VAC/references.html#klee_1995http://www.carla.umn.edu/assessment/VAC/references.html#klee_1995http://www.carla.umn.edu/assessment/VAC/references.html#hughey_1983http://www.carla.umn.edu/assessment/VAC/gloss_windows/references.html#POLIAhttp://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#taggart_1998


6/14

6

ORGANIZATION 20 POINTS POSSIBLE


20 - 18

Excellent to Very Good.well-framed andorganized (with clear introduction, conclusion).coherent .succinct .cohesive (excellent useof connective words)

17 - 14

Good to Average adequate, but looseorganization with introduction and conclusion(though they may be limited or one of the twomay be missing) .somewhat coherent .morewordy rather than succinct .somewhatcohesive (good use of connective words)

13 - 10

Fair.lacks good organization (no evidenceof introduction, conclusion) .ideas may bedisconnected, confused .lacks coherence.wordy and repetitive .lacks consistent useof cohesive elements

9 - 7

Poor.confusing, disconnected

organization .lacks coherence, so much sothat writing is difficult to follow .lackscohesion

LANGUAGE USE/GRAMMAR/MORPHOLOGY 25 TOTAL POINTS POSSIBLE


25 - 22

Excellent to Very Good.great variety ofgrammatical forms (e.g., range of indicativeverb forms; use of subjunctive) .complexsentence structure (e.g., compoundsentences, embedded clauses) .evidence of"Spanish-like" construction .mastery ofagreement (subj/verb; number/gender) .veryfew errors (if any) overall with none that

obscure meaning

21 - 18

Good to Average.some variety ofgrammatical forms (e.g., attempts, though notalways accurate, of range of verb forms, useof subjunctive) .attempts, though not alwaysaccurate, at complex sentence structure (e.g.,compound sentences, embedded clauses).little evidence of "Spanish-like" construction,though without clear translations from English.occasional errors with agreement .someerrors (minor) that dont obscure meaning

17 - 11

Fair.less variety of grammatical forms(e.g., little range of verb forms; inaccurate, if

any, attempts at subjunctive) .simplisticsentence structure .evidence of "English-like"construction (e.g., some direct translation ofphrases) .consistent errors (e.g., withagreement), but few of which may obscuremeaning

10 - 5

Poor.very little variety of grammaticalforms .simplistic sentence structure thatcontains consistent errors, especially withbasic aspects such as agreement .evidence oftranslation from English .frequent andconsistent errors that may obscure meaning

VOCABULARY/WORD USAGE 20 TOTAL POINTS POSSIBLE


20 - 18

Excellent to Very Good.sophisticated,academic range .extensive variety of words.effective and appropriate word/idiom choice

and usage .appropriate register

17 - 14

Good to Average.good, but notextensive (less academic), range or variety

.occasional errors of word/idiom choice orusage (some evidence of invention of "false"cognates), but very few or none that obscuremeaning .appropriate register

13 - 10Fair.limited and "non-academic" range(frequent repetition of words) .moreconsistent errors with word/idiom choice or


7/14

7

usage (frequent evidence of translation;invention of "false" cognates) that may(though seldom) obscure meaning .someevidence of inappropriate register

9 - 7

Poor.very limited range of words.consistent and frequent errors with

word/idiom choice or usage (ample evidenceof translation) .meaning frequently obscured.evidence of inappropriate register

MECHANICS 5 TOTAL POINTS POSSIBLE


5

Excellent to Very Good.demonstratesmastery of conventions .few errors in

spelling, punctuation, capitalization, and useof accents

4

Good to Average.occasional errors in

spelling, punctuation, capitalization, and useof accents, but meaning is not obscured

3

Fair.frequent errors in spelling,punctuation, capitalization, and use ofaccents that at times confuses or obscuresmeaning

2

Poor.no mastery of conventions

.dominated by errors in spelling,punctuation, capitalization, and use ofaccents

Total Score_____ COMMENTS:

Figure F presents an adaptation of a well-known analytic scale for evaluating ESL writing performance.

Describing this rubric,Tedick (2002)writes: "Note that the scale assigns different weights to different

features. This allows a teacher to give more emphasis to content than to grammar or mechanics, for

example. The option to weigh characteristics on the scale represents an advantage to analytic

scoring." (p. 35).

Figure F2 shows an analytic scale for role plays and interviews used with students in first-year French

courses at the University of Minnesota. This rubric can be used with other languages. In this example,

all criteria are weighted equally.

Analytic rubric for role plays and interviewsPost-secondary, Year 1

Communicative Success (Would a listener accustomed to the speech of learnersunderstand?)

A 6 / 5.5 Understand all of the message.

A- 5 Understand the general message and most of the details.

B 4.5 Understand general message, but only some of the details.

C 4Have some idea of the general message, but would not be sure to haveunderstood.

D-F 3.5 - 0 Do not understand what the speaker is trying to say.

Pronunciation & Fluency

A 6 / 5.5Speech is smooth; speaker is comfortable and confident in use of the language. Nomispronunciation that would interfere with comprehension by a sympathetic nativespeaker.

A- 5Speech is occasionally hesitant; some rephrasing. Mispronunciation causingmisunderstanding occurs only rarely.

B 4.5Speech is hesitant (e.g. frequent rephrasing, sentences left unfinished, longpauses). Several misunderstandings arise from mispronunciation of words orerrors in intonation.

C 4Speech hesitant and choppy; conversation is almost impossible. Mispronunciationand inaccurate stress make understanding difficult. Has to repeat a lot to beunderstood; OR not enough speech to evaluate.

D-F 3.5 - 0 Speech limited to isolated words, or mispronunciation makes comprehensionimpossible.
http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002


8/14

8

Vocabulary

A 6 / 5.5Shows control of a wide range of the vocabulary taught in class and always usesthis vocabulary appropriately.

A- 5Shows control of an adequate range of the vocabulary taught in class and mostoften uses this vocabulary appropriately.

B 4.5 Some control of new vocabulary, but relies on fixed expressions/basic vocabularyor uses vocabulary inappropriately.

C 4Shows very limited control of the vocabulary taught, making discussion of relatedtopics extremely difficult; OR not enough speech to evaluate.

D-F 3.5 - 0 Shows no command of the vocabulary taught, making communication impossible.

Grammar

A 6 / 5.5Shows consistent control of the structures taught in class and communication isnever impeded.

A- 5 Usually controls structures taught in class.

B 4.5 Shows partial control of structures taught in class.

C 4 Speech is very difficult to understand due to lack of control of structures taught;OR not enough speech to evaluate.

D-F 3.5 - 0 Extreme lack of control of structures taught in class.

Role Plays/Interviews(Does it sound like a real conversation?)

A 6 / 5.5Exchange is well-connected and appropriate to the topic and situation. Amount oftime spent conversing is appropriate for the task assigned and the topic isadequately covered.

A- 5 Exchange is usually well-connected and appropriate to the topic and situation.

B 4.5

Some misunderstandings occur because discourse is not sufficiently connected orconversation is not always appropriate to the topic and situation; or speaker(s)does not maintain conversation for assigned length of time and needs to be told to

continue.

C 4Misunderstandings frequently occur between participants because discourse is notconnected; or conversation is often inappropriate to topic or situation.

D-F 3.5 - 0Exchange is not connected (many non-sequiturs; speaker unable to hold uphis/her end of the conversation); or conversation is entirely inappropriate to topicor situation.

Department of French and Italian, College of Liberal Arts, University of Minnesota

There are more sample analytic rubrics in the Evaluation > Examplessection.

Advantages:

They provide useful feedback to learners on areas of strength and weakness.

Their dimensions can be weighted to reflect relative importance.

They can show learners that they have made progress over time in some or all dimensions

when the same rubric categories are used repeatedly (Moskal, 2000).

Disadvantages:

"The whole is greater than the sum of its parts."Tedick (2002)notes: "Separate scores fordifferent aspects of a students writing or speaking performance may be considered artificial in

that it does not give the teacher (or student) a good assessment of the "whole" of a

performance." (p. 36).

They take more time to create and use.

There are more possibilities for raters to disagree. It is more difficult to achieve intra- and

inter-rater reliability on all of the dimensions in an analytic rubric than on a single score

yielded by a holistic rubric.

There is some evidence that raters tend to evaluate grammar-related categories more harshly

than they do other categories (McNamara, 1996), thereby overemphasizing the role of

accuracy in providing a profile of learners' proficiency.
http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#mcnamara_1996http://www.carla.umn.edu/assessment/VAC/references.html#mcnamara_1996http://www.carla.umn.edu/assessment/VAC/references.html#mcnamara_1996http://www.carla.umn.edu/assessment/VAC/references.html#mcnamara_1996http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#moskal_2000


9/14

9

There is some evidence that "when raters are asked to make multiple judgments, they really

make one..."(Fulcher, 2009).Care must be taken to avoid a "halo effect" and focus on the

individual criteria to assure that diverse information about the learner's performance is not

lost.

ProcessTypes of rubrics: Primary Trait and Multiple Trait

Primary traitscoring, as developed by Lloyd-Jones and Carl Klaus (Lloyd-Jones, 1977)was designedto evaluate the primary language function or rhetorical trait elicited by a given writing task or prompt.

"Primary trait assessment in its initial formulations focused on the specific approach that a writer

might take to be successful on a specific writing task; every task required its own unique scoring

guide" (Applebee, 2000,p. 4). In its original form, primary trait scoring would be strictly classified as

task-specific, and performance would be evaluated on only one trait, such as the "Persuading an

audience" example fromTedick(2002, p. 36) for a task requiring learners to write a persuasive letter

to the editor of the school newspaper:

Fig. Fx. Primary Trait: Persuading an audience

0 Fails to persuade the audience.

1 Attempts to persuade but does not provide sufficient support.

2Presents a somewhat persuasive argument but without consistent development

and support

3 Develops a persuasive argument that is well developed and supported.

Today, you may find that primary trait rubrics vary markedly from their original design and intended

use.Applebeenotes: "Over the years as primary trait approaches were used more widely, they

evolved into a more generic approach which recognized the similarities in approach within broad uses

or purposes. The basic question addressed in scoring, however, remained, 'Did the writer successfully

accomplish the purpose of this task?' To insure that raters maintained this focus, scoring guidelines

usually instructed raters to ignore errors in conventions of written language, and to focus on overall

rhetorical effectiveness" (p. 4). Primary trait scoring can be used with speaking tasks as well as with

assessments of the interpersonal and presentational modes.

If you search the Web for primary trait rubrics, you will occasionally find examples that include several

traits rather than the one main criterion for successful communication within a specified rhetorical or

functional domain (e.g.,SUNY Oswego, Which type of rubric is best? Fig. 3). In the Virtual Assessment

Center, we adopt the distinctions outlined byTedick(2002) and refer to task-specific scoring grids

with more than one dimension as multiple trait, or multitrait, rubrics.

When would you use primary trait rubrics in the classroom? They provide minimal feedback to

learners, and it probably would not be fair to base important decisions like grades on whether or not

learners perform well on just one criterion. One scenario might be to use primary trait rubrics in

formative assessments designed to determine how well learners perform a particular language

function they have been working on in class. For example, if several lessons have been devoted to

working on descriptive language, a culminating writing task might be scored solely on its effectiveness

as a description.

Multiple trait rubrics.Hamp-Lyons(1991) coined the term multiple trait scoringfor rubrics that she

designed, based on the concepts of primary trait scoring, to provide diagnostic feedback to learners

and other stakeholders about performance on "context-appropriate and task-appropriate criteria" for a

specified topic/text type. She designed her multiple trait rubrics to be applicable across a range of

similar tasks. Currently, multiple trait (or multitrait) rubrics are commonly considered to be task-

specific, although one or more of their dimensions might also be found in generic, analytic rubrics.

Many examples of rubrics of this type that you may find on the Web or in other resources often

accompany a given task, and may not be readily applicable to other tasks without adaptation. Figure

Fy illustrates a task and multitrait scoring rubric from a resource for language teachers (Petersen,

1999).
http://languagetesting.info/features/halorating/rating.htmlhttp://languagetesting.info/features/halorating/rating.htmlhttp://languagetesting.info/features/halorating/rating.htmlhttp://www.carla.umn.edu/assessment/VAC/references.html#lloyd_1977http://www.carla.umn.edu/assessment/VAC/references.html#lloyd_1977http://www.carla.umn.edu/assessment/VAC/references.html#lloyd_1977http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.oswego.edu/~shindler/typesofrubrics.htmhttp://www.oswego.edu/~shindler/typesofrubrics.htmhttp://www.oswego.edu/~shindler/typesofrubrics.htmhttp://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#hamp_1991http://www.carla.umn.edu/assessment/VAC/references.html#hamp_1991http://www.carla.umn.edu/assessment/VAC/references.html#hamp_1991http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#petersen_1999http://www.carla.umn.edu/assessment/VAC/references.html#hamp_1991http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.oswego.edu/~shindler/typesofrubrics.htmhttp://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#tedick_2002http://www.carla.umn.edu/assessment/VAC/references.html#applebee_2000http://www.carla.umn.edu/assessment/VAC/references.html#lloyd_1977http://languagetesting.info/features/halorating/rating.html


10/14

10

Figure Fy. Task and multiple trait mini-rubric*

Activity Description:

If language teachers were paid every time they had to remind students to speak the targetlanguage (TL), they could probably retire.

With this in mind, you will be a part of a group that will meet at the beginning of each class(groups will change from time to time). Your teacher will have an activity prepared for thegroups to complete each day (this gives your teacher the opportunity to take attendance andfinish all those other time-consuming tasks).

Each group should elect a leader to take charge and keep the group on task, making sure thateveryone participates. Always speak as much TL as possible during your warm-up. Help andlearn from each other. Use the TL to greet and say your farewells to your group members at theend of the activity.

Primary Activity Standard: Communication Standard 1.1 (InterpersonalCommunication)

Students engage in conversations, provide and obtain information, express feelings andemotions, and exchange opinions.

Excellent Average Needs Work

Time on Task

The group formsimmediately to work onactivity until the teacherindicates otherwise; ifgroup finishes early,members discuss topicsrelated to TL.

10 9

The group forms fairlysoon to work mostly onactivity until the teacherindicates otherwise; ifgroup finishes early,members are either silentor discuss topics notrelated to TL.

8 7 6

The group takes a longtime to form; they do notwork on activity (unlessthe teacher walks by); ifgroup finishes early,members discuss topicsnot related to TL.

5 4 3 2 1 0

Participation

All group membersparticipate equallythroughout the entireactivity.

5

All group members butone participate equallythroughout the activity.

4 3

More than one groupmember does notparticipate equallythroughout the activity.

2 1 0

Group

Cooperation

All members cooperate

to help each other learn;if anyone has been

absent, the group helpshim/her; no one acts"superior."

10 9

Most members cooperate

to help each other learn; ifanyone has been absent,

the group sometimes helpshim/her; no one acts"superior."

8 7 6

Members do notcooperate to help each

other learn; if anyonehas been absent, the

group does not help;some members act"superior."

5 4 3 2 1 0

Use of TL

Members use as much TLas possible (also to greetand say farewells).

5

Members use some TLduring activity (also togreet and say farewells).

4 3

Members rarely use TLduring activity (neitherdo they greet nor sayfarewells).

2 1 0

1999 Wade Petersen

Multiple trait rubrics look like analytic rubrics in that performance is evaluated in several categories,

and, in practice, you may find the terms used interchangeably. However, analytic rubrics usually

evaluate the more traditional and generic dimensions of language production, while the criteria in

multiple trait rubrics focus on specific features of performance necessary for successful fulfillment of a

given task or tasks.


11/14

11

Advantages

The rubrics are aligned with the task and curriculum.

Aligned and well-written primary and multiple trait rubrics can ensure construct and content

validity of criterion-referenced assessments.

Feedback is focused on one or more dimensions that are important in the current learningcontext.

With a multiple trait rubric, learners receive information about their strengths and

weaknesses.

Primary and multiple trait rubrics are generally written in language that students understand.

Teachers are able to rate performances quickly.

Many rubrics of this type have been developed by teachers who are willing to share them

online, at conferences, and in materials available for purchase.

Disadvantages

Information provided by primary trait rubrics is limited and may not easily translate into

grades.

Task-specific rubrics cannot be applied to other tasks without adaptation of at least one or

more dimensions.

Process: Creating rubrics

There are many rubric resources available to teachersonline and in published materialsso the

first piece of advice we have to offer is: Find and adapt existing rubrics! It is rare to find a rubric

that is exactly right for your situation and your students, but by using rubrics that have workedwell for others as a starting point, you can save a great deal of time.

There are many rubric formats. In the grid format shown here, which is one of the possible ways

to lay out a rubric, we illustrate a few common, frequently recommended, features of multiple

trait rubrics:

An even number (4 or 6) of levels of performance on the scale. When there is an odd

number of levels, the middle level tends to become a catch-all category. With an even

number of levels, raters have to make a more precise judgment about a performance

when its quality is not at the top or bottom of the scale.

High to low scale. In the graphic, the highest level of performance is described at theleft. Students read first the description of an exemplary performance in each criterion. A

few labels for a four-point scale include:

4 3 2 1

Exemplary Excellent Acceptable Unacceptable

Exceeds expectations Meets expectations Progressing Not there yet

Superior Good Fair Needs work

Limited number of dimensions or criteria. The criteria are those components that are

most important to evaluate in the given task and instructional context. A rubric with too

many dimensions may be unworkable in classroom assessment.

Equal steps along the scale. The difference between 4 and 3 should be equivalent to the

difference between 3 - 2 and 2 - 1. "Yes, and more", "Yes", "Yes, but", and "No" are

ways for the rubric developer to think about how to describe performance at each scale

point. Some common descriptive terms are listed in the chart below.


12/14

12

4 3 2 1

Task

requirements

All Most Some Very few or

none

Frequency Always Usually Some of the

time

Rarely or not

at all

Accuracy No errors Few errors Some errors Frequent errors

Comprehensibility Always

comprehensible

Almost always

comprehensible

Gist and main

ideas are

comprehensible

Isolated bits

are

comprehensible

Content coverage Fully

developed,

fully supported

Adequately

developed,

adequately

supported

Partially

developed,

partially

supported

Minimally

developed,

minimally

supported

Vocabulary

Range

Variety

Broad

Highly varied;

non-repetitive

Adequate

Varied;

occasionally

repetitive

Limited

Lacks variety;

repetitive

Very limited

Basic,

memorized;

highly

repetitive

Considerations for Task-Based Rubric Development*

Getting Started: Determining the criteria that will be valued for a particular task.

1. Brainstorm all the possible elements or criteria that could be assessed in the

performance task.

2. Determine which elements are "non-negotiable." Which criteria could be part of the task

description as baseline requirements or provided as a checklist?

3. Prioritize the elements that are left:

o What are the content and language goals of the unit?

o What do you really want the students to emphasize in their performance?

o How important is the overall "look" of the project (interest, appeal, creativity, neatness)?

o Is culture represented in the rubric, if applicable?

o Are the standards you targeted represented in the rubric?

4. Determine 3 to 5 elements or criteria that will be incorporated in the rubric to define a

quality performance. In trying to be thorough, an unwieldy rubric may be constructed,

with so many elements being assessed that the rubric is time-consuming to fill out or

oral performances will have to be taped (audio or video) in order to repeat them severaltimes for the purposes of assessment.

Considering the Levels of the Rubric: Determining the number of levels and defining

them.

1. How many levels of performance do you wish to include in the rubric? How should they

be defined? For example, "does not meet expectations," "meets expectations," "exceeds

expectations." Or you may choose to use simply a 3, 4, or 5-point scale, noting,

however, that a 3-point scale does not account for the fluctuation that exists within the

average range. Some suggest that a 4-point scale is ideal, and that more than 4 points

makes a scale cumbersome and difficult to use.
http://www.carla.umn.edu/assessment/VAC/Evaluation/tbrd.htmlhttp://www.carla.umn.edu/assessment/VAC/Evaluation/tbrd.html


13/14

13

2. Consider the elements or criteria you have chosen one at a time. Begin with the highest

level of the scale to define top quality performance. This is the level that you want all

students to achieve and it should be challenging. How would you describe a

representation that exceeds expectations? Meets expectations? Does not meet

expectations?

3. Are the levels you have created parallel? That is, are the criteria present in all levels?4. Is there continuity in the difference between the criteria for exceeds vs. meets, and

meets vs. does not meet expectations? The difference between a 2 and a 3 performance

should not be more than the difference between a 3 and a 4 performance.

5. Do the levels reflect variants in quality and not a shift in importance of the criteria?

6. Is there an expectation of quality at the average (meets expectations) level of the scale?

Other issues to consider for rubric creation:

Are the characteristics of each performance level described clearly? Will students be able

to self-assess with the descriptors given? Will the descriptors give students enough

information to know what they need to improve?

Does the rubric adequately reflect the range of levels at which students may actually

perform given tasks?

Are the criteria at each level defined clearly enough to ensure that scoring is accurate,

unbiased and consistent? Could several teachers use the rubric and score a students

performance within the same range?

Does the rubric attend to process as well as product?

Are all criteria equally important, or does it make sense to weight an element more than

the others?

Are you attending carefully to the language used in the rubric? Use demonstrative verbs.

Keep to observable behaviors. Avoid negatives ("begins without preparation" vs. "doesnot prepare"). Be specific. Instead of "many errors" you may want to specify "six or

more errors". At the same time, be sure the rubric is generally qualitative in nature

rather than quantitative.

Other issues to consider when using rubrics:

Rubrics need to be piloted or field tested.

Rubrics need to be discussed with students to create an understanding of expectations;

you cannot write a paragraph defining each word in the rubric.

Are the criteria at each level defined clearly enough to ensure that scoring is accurate,

unbiased and consistent? Could several teachers use the rubric and score a studentsperformance within the same range?

There is a fine balance between modeling excellent work and creating a "template" that

is replicated by the students ad nauseam to the detriment of creativity.

"If a student can achieve a high score on all the criteria and still not perform well at the

task, you have the wrong criteria" (Wiggins, cited in Clementi, 1999).

Consider whether a rubric needs revision for a specific task. Do some of the criteria on

the rubric go beyond this particular performance (that is, if youve created a rubric that

is more "generic" and can be used for many tasks over time)?

Make sure that the expectations in the rubric are directly aligned with the instruction of

the lesson/unit. Students shouldnt be expected to do what they havent been previously

taught or shown.

Some suggest that generic rubrics are more useful because creating rubrics is time-

consuming and the more often they can be applied, the better. It is also more

informative for students if the same rubric is used again and again, because they can

see themselves making progress over time. On the other hand, generic rubrics are much

less tied to the task and are not able to provide criteria for specific language use

expectations or content knowledge.


14/14

14

Process: Grades

While we use rubrics to judge how well how well learners have met appropriate and clear

expectations set for them and to give them feedback about their performance along a quality

continuum, we also know that generally we are expected to issue grades as a descriptor of

student performance.

Teachers often construct their own scales for converting rubrics to grades. As with rubrics,

students need to be clearly informed of what the lowest "acceptable" score would be. The

Fairfax County (VA) Public Schools World Languageweb site provides one model of a formula

for converting scores from a point-based analytical rubric to a grade. ChoosePerformance

Assessments for Language Students (PALS). Under each level, you will see a conversion

chart for each rubric and task type listed.

The assessment section of theNebraska Foreign Language Frameworks, available in PDF

format, also features an assessment score conversion chart for converting a series of raw

scores to percentage scores. The chart is found on the last page of the pdf document.
http://www.fcps.edu/is/worldlanguages/index.shtmlhttp://www.fcps.edu/is/worldlanguages/index.shtmlhttp://www.fcps.edu/is/worldlanguages/pals/index.shtmlhttp://www.fcps.edu/is/worldlanguages/pals/index.shtmlhttp://www.fcps.edu/is/worldlanguages/pals/index.shtmlhttp://www.education.ne.gov/forlg/FrameworksMain.htmhttp://www.education.ne.gov/forlg/FrameworksMain.htmhttp://www.education.ne.gov/forlg/FrameworksMain.htmhttp://www.education.ne.gov/forlg/FrameworksMain.htmhttp://www.fcps.edu/is/worldlanguages/pals/index.shtmlhttp://www.fcps.edu/is/worldlanguages/pals/index.shtmlhttp://www.fcps.edu/is/worldlanguages/index.shtml

carla rubrics

Documents