Cultural Validity in Assessment Practices

Guillermo Solano-Flores

American Institutes for Research

Mid-Atlantic Equity Center Annual Regional Conference, Washington, DC,

March 12, 2004

Paradigms in testing

Traditional:

• Classifies individuals into large categories

• Focuses on differences between groups

• Uses adaptation and accommodation of tests

• Centralized

• Deficit model view

Alternative:

• Defines a socio-cultural context

• Focuses on score dependability

• Emphasizes the process of test development

• Community-based

• Multidisciplinary perspective

Overview

• The need for interdisciplinary approaches to test design

• Understanding how culture and language influence test taking

• Implications: new paradigms in testing

First NSF study: Assessing assessment cultural validity

• Participants: Students from twelve sociocultural contexts (ethnicity, language background, SES, geographical area, locale, type of school)

• Students were given items used in standardized testing that are supposed to be “culturally-sensitive”

• Students were asked to explain how they interpreted the items and how they related their content to their personal experience

Second NSF study: Sociolinguistic perspectives in

testing • Participants: Students from different geographical

areas who are assumed to speak different dialects of the same language; teachers who are familiar with the dialect spoken by their students

• Teachers from different sites adapt the same set of items based on their knowledge of their students’ dialects

• Students take the tests in both dialect versions

Pressing issues in the testing of linguistic and cultural

minorities

• Accountability based on standardized test scores

• Bilingual and multicultural education under attack

Limitations of current approaches to testing linguistic

and cultural minorities

• Lack of theories of language and culture• Testing practices driven by erroneous

assumptions about language and culture• Erroneous assumptions about the

effectiveness of current testing practices• Recent, cognitive approaches to testing

overlook the important cultural influences on cognition

Examining the linguistic demands posed by test items

A measurement of 60 inches is equal to how many feet?

S

NP VP

Det N PP

A

mea

sure

men

t

P NP

of Det N

60 inches

AdjP NP

is Adj Prt AdjP N

equal to Adv Adj feet

how many

V

The Lunch Money item

Sam can purchase his lunch at school. Each day he wants to have juice that costs 50¢, a sandwich that costs 90¢, and fruit that costs 35¢. His mother has only $1.00 bills. What is the least number of $1.00 bills that his mother should give him so he will have enough money to buy lunch for 5 days?

What is the least number of $1.00 bills that his mother should give him so he will have

enough money to buy lunch for 5 days? S

NP VP

N V

PP N AdjP Det

NP

of -meemenment

least -meemenment

the -meemenment

is -meemenment

What -meemenment

Mod

P NP

Det

Adj

nu

mb

er

S’ N

one -meemenment

bills -meemenment dollar

-

N Ccl S

that -

NP

Det N

Ø NP

N

S’ NP V Aux

VP

his -meemenment

he -meemenment

so -meemenment

give -meemenment m

oth

er

shou

ld

V Aux N

VP NP

S Cs

him -meemenment

have -meemenment

will -meemenment Adj

S’ N AdjP

NP

enou

gh m

oney

S Ø comp

VP

V

N Det

NP P

NP

N PP

IM

to -meemenment

days -

5 -

for -meemenment

lunch -meemenment

buy -meemenment

Major (yet preliminary) finding:

The syntactical structure of some test items is unnecessarily complex

Understanding sociocultural influences on test taking

Assessing others: The Kayak experience

A

C

B

H

D

F

E

G

Larry’s body measures

Henry’s body measures

Socio-cultural activity questions

How do you see [this item] as part of...

1. …what you do when you are not at school?

2. …what you do for fun when you are at school?

3. ...your school day in the classroom?

4. ...any traditions that you have?

World views: a low-income student’s response to the Lunch

Money item

Interviewer: What is this item about?

Student: It’s about Sam, trying to get her lunch, but her mom only has one dollar, and she needs more for five days, so I think she should give her a dollar ninety-five.

Cultural factors relevant to assessment

• Epistemologies -- ways of constructing knowledge and making sense of experience

• Teaching and learning styles -- ways of transmitting and acquiring knowledge

• Discourse styles -- ways of expressing ideas

• interpret what a test item is about

• use their knowledge and experience to solve problems

• demonstrate their knowledge

Cultural influences in assessmentDifferent cultural backgrounds produce different ways in which students:

Understanding the students’ linguistic proficiency

Language dominance: Traditional, simplistic view

Native language Mode English

Non-Bilingual Native English Speaker

R W L S Fully Bilingual

R W L S

English-Language Learner R W L S

100%

0

100%

Proficiency

Patterns of language dominance diversity: A more realistic view

Native language Mode English ELL Student 1

R W L S ELL Student 2

R W L S ELL Student 3 R W L S

100% 0 100%

Proficiency

• Underestimating students’ English proficiency

• Overestimating students’ proficiency in their native language

• Lowering academic standards and expectations

Assessing linguistic minority students: Common mistakes

Dimensions of item design and review: data sources, methods

Formal text of the items

Pragmatic practitioner reviews, focus groups, questionnaires, review checklists

Individual student talk-alouds, student read-alouds, interviews, student inferred interpretation of items

Differential group measures, p values, mean scores

Dimensions of item design and review: focus

Formal Linguistic complexity

Pragmatic Cultural appropriateness, dialect, fairness

Individual Cognitive processes, sociocultural influences, student epistemology

Differential Statistical significance of group differences

Difficulty reading “$1.00 bills” in “His mother has only $1.00 bills.”

Identified by linguists: Yes

Anticipated by at least 20% of teachers: No

Observed in at least 20% of students: Yes

Statistically significant differences

between groups: Yes

Difficulty reading “least” in“What is the least number

of $1.00 bills…?”

Identified by linguists: Yes

Anticipated by at least 20% of teachers: Yes

Observed in at least 20% of students: No

Statistically significant differences between groups: No

Why we should pay more attention to the process of test

development

Validity and high-stakes testing

Information on the validity of a test for a given population of students is usually available after decisions affecting those students and based on the scores from that test have been made.

Validity and high-stakes testing

Language: Tolerance to error

What about English-language learners?

Inadequate test adaptation: An example

Printed Spanish takes about 25 percent more characters than printed English. To deal with this difference, some test publishers reduce the font size of the text and the margins in the Spanish version. Where did standardization go?

El español impreso emplea cerca de un 25 por ciento más de caracteres que el inglés impreso. Para resolver tal diferencia, algunas compañías que publican pruebas utilizan un tipo de letra más pequeño de letra en el texto y reducen los márgenes de la versión en español. ¿Qué pasó con la estandarización?

Thinking about language 1. Literal equivalence across

languages Is technical terminology accurate in both languages?

2. Appropriateness of language to a broad cultural group

Should decimal metric system units be used in the exercises in Spanish?

3. Appropriateness of language to a sociocultural context

Are these students more familiar with kilograms and grams (which are used in their home countries) or with pounds and ounces (which are part of their everyday life experience in the U.S.)?

4. Correspondence between item structure and discourse patterns

Should the sequence of the item components (e.g., contextual information, table with numeric information, space for computations) be the same for both languages, or should the sequence be different for each language?

Disciplines relevant to test development

Traditional:• Psychometrics• Cognitive psychology

Other:• Cultural

anthropology• Sociolinguistics• Structural linguistics• Reading

Conclusions

Current approaches to testing English language learners do not address effectively the fact that assessments are extremely sensitive to wording

Implications for assessment

• New paradigms in testing

• Multidisciplinary approaches

• Combining quantitative and qualitative methods

Workshop exercises

Exercise 1: Gumball machine

The gum ball machine has 100 gum balls; 20 are yellow, 30 are blue, and 50 are red. The gum balls are well mixed inside the machine.

Jenny gets 10 gum balls from this machine. What is your best prediction of the number that will be red? 

Exercise 2: Metals

Many things are made of metal, such as pots, pans, tools, and wire. Give two reasons why metals are used to make many different things.

Exercise 3: Mountains

The pictures below show the same river and mountains, but one picture shows how they looked millions of years ago, and the other picture shows how they look now. Circle the letter under the picture that shows how the river and mountains look now. Explain how you can tell this.

References (1)• Solano-Flores, G., & Trumbull, E. (2003). Examining language

in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2), 3-13.

• Solano-Flores, G. (2003). The multidimensionality of test review and test design: A conceptual framework for addressing linguistic and cultural diversity in testing. Paper presented at the !0th Biennial Conference of the European Association for Research on Learning and Instruction, Padova, Italy - August 26 – 30, 2003.

• Solano-Flores, G., Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities. International Journal of Testing, 2(2), 107-129.

References (2)

• Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38(5), 553-573.

• Solano-Flores, G., Lara., J., Sexton, U., & Navarrete, C. (2001). Testing English language learners: A sampler of student responses to science and mathematics test items. Washington, DC: Council of Chief State School Officers.