validation of score meaning and justification of a score use: a comprehensive model of defensible...

Validation of Score Meaning and Justification of a Score

Use:

A Comprehensive Model of Defensible Testing Practice

Triangle Assessment Networking Group22 February 2017

Gregory J. CizekGuy B. Phillips Distinguished Professor of Educational Measurement and

EvaluationSchool of Education, University of North Carolina-Chapel Hill

[email protected]

Overview

I. Some FoundationsII. A Major Flaw in Current Validity TheoryIII. A Revised FrameworkIV. Questions, Comments...

Some Foundations* Measurement instruments are integral

to training, practice, development and research in the social sciences.

3

Some Foundations* Measurement instruments are integral

to training, practice, development and research in the social sciences.

* The quality of the data yielded by these instruments is the focus of measurement specialists and an essential concern of test users and consumers of test information.

4

Some Foundations (cont’d)

Professional standards applicable to these instruments have existed for over 50 years:

*Technical Recommendations for Psychological Tests and Diagnostic Techniques (APA, 1954)

* Standards for Educational and Psychological Testing (AERA, APA, NCME, 2014)

5

Validity is important.

“one of the major deities in the pantheon of the psychometrician”

(Ebel, 1961, p. 640)

6

Validity is important.

“one of the major deities in the pantheon of the psychometrician” (Ebel, 1961, p. 640)

“the most fundamental consideration in developing tests and evaluating tests” (AERA, APA, NCME, 2014, p. 11)

7

Validity has problems.

“For a concept that is the foundation of virtually all aspects of our measurement work, it seems that the term validity continues to be one of the most misunderstood or widely misused of all.”

(Frisbie, 2005, p. 21)

8

9

The Canon

1. Validity pertains to score inferences.

“I don’t want to know which questions you answered correctly. I want to know how much....you know. I need to leap from what I know and don’t want to what I want but can’t know. That’s called inference.”

(Wright, 1994)

10

The Canon (cont’d)

2. Validity is not a characteristic of a test.

“One validates, not a test, but an interpretation of data arising from a specified procedure.”

(Cronbach, 1971, p. 447)

11


3. Validity is a unitary concept; all validity is construct validity.

“What is singular in the unified theory is the kind of validity: All validity is of one kind, namely, construct validity.” (Messick, 1998, p. 37)

12


4. Validity is a matter of degree.

“Validity statements are not dichotomous (valid/ invalid) but rather are described on a continuum.”

(Zumbo, 2007, p.

50)

13


5. Validation involves gathering and evaluating evidence bearing on intended test score inferences.

“Validity is an integrated, or unified, evaluation of the [score] interpretation.”

(Kane, 2001, p. 329)

14


6. Validation is an on-going endeavor.

“Every treatise on the topic [indicates that] construct validation is a never-ending process.”

(Shepard, 1993, p. 407)

15


Five Sources of Validity Evidence:Evidence based on...

1) test content2) response process3) internal structure4) relations to other variables5) consequences of testing

A Major Flaw

“Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores....”

(Messick, 1989, p. 13, emphasis added)

Two Problems:

17

Two Problems:

1. conflates validity (a property) with validation

(a process)

18

Two Problems:

1. conflates validity (a property) with validation (a process)

2. requires integration of that which cannot be integrated

19

Is it even possible?….

* “Validity for a specific test use” is nonsensical:

Once evidence bearing on the intended score interpretation has been gathered, the validity—that is, the meaning of scores—does not change depending on the consequences of test use, or whenever a new use is envisioned.

20

Is it even possible?….* The value of evidence based on most of the currently accepted sources of validity is straightforward; however, evidence based on consequences is not unequivocally interpretable as positive or negative.

21


* Evidence about the extent to which a test yields accurate inferences and information about real or potential consequences of using the test are not compensatory in any logical sense.

22


* Evidence about the extent to which a test yields accurate inferences and information about real or potential consequences of using the test are not compensatory in any logical sense.

* Any attempted integration confounds conclusions about both the scientific meaning of the score and the desirability of using the test. 23

Example

* Consider a total raw score on a test comprising 20 items measuring French vocabulary and 20 items measuring Geometry.

* An overall score could be calculated, but what would be a defensible interpretation of a total score of, say, 30?

24

The Most Telling Finding:

In over 20 years, no example of Messick’s proposed outcome (i.e., a synthesis of theoretical, empirical, and social consequences data yielding an overall, integrated judgment about validity) has ever been produced.

Validation practice is the victim.

25

First: An Improved Definition

“Validity is the degree to which scores on an appropriately administered instrument support inferences about variation in the characteristic that the instrument was developed to measure.”

(Cizek, 2012, p. 5)

26

Second: A Comprehensive Framework for

Defensible Testing

27

A Comprehensive Framework for Defensible Testing

Validation of Score Meaning

“What is the evidence that the score can be interpreted as intended?”

28


Validation of Justification Score Meaning of Score Use

“What is the evidence that “What is the evidence that the score can be interpreted the score should be used as intended?” as proposed?”

29


Validation of Justification Score Meaning of Score Use

“What is the evidence that “What is the evidence that the score can be interpreted the score should be used as intended?” as proposed?”

30

How theStandards for Educational and

Psychological TestingMust Change

31

The Standards for Educational and Psychological Testing

Evidence based on...

32



1) Test Content

33



1) Test Content

2) Response Process

34



1) Test Content

2) Response Process

3) Internal Structure

35



1) Test Content

2) Response Process


4) Relations to Other Variables

36



1) Test Content

2) Response Process



5) Consequences of Testing37



1) Test Content

2) Response Process



5) Consequences of Testing38


Standards for Justification Educational and of Score Use Psychological Test “What is the evidence that Use the score should be used

as proposed?”

39

Standards for Educational and Psychological Test Use

A beginning outline of sources of evidence

40

Sources of Evidence for Justifying Test Use

Examples

Evidence Based on Consequences of Testing

* Evaluation of anticipated benefits* Consideration of negative consequences (e.g., opt-out, curriculum narrowing, etc.)* Consideration of false positive/ negative rates

Evidence Based on Costs of Testing* Overall cost of testing* Cost-benefit evaluation* Consideration of opportunity costs (e.g., instructional time)

Evidence Based on Alternatives to Testing

* Evaluation of relative value of alternative testing methods, formats, or procedures* Evaluation of non-test alternatives to accomplish intended goals

Evidence Based on Fundamental Fairness

* Evaluation of stakeholder inclusion* Investigation of opportunity to learn* Provision of due notice* Examination of differential impact across relevant subpopulations



1) Test Content

2) Response Process



42



1) Test Content

2) Response Process



43



1) Test Content

2) Response Process

3) Hypothesized Relationships among Variables

44



1) Test Content

2) Response Process

3) Hypothesized Relationships among Variables

4) Test Development and Administration Procedures

45

Sources of Evidence Supporting Score Meaning

Source

Examples

Evidence Based on Test Content

* Grounding in relevant theoretical dimensions or relationships* Job analyses* Content/curricular alignment studies

Evidence Based on Response Processes

* Cognitive labs* Think-aloud protocols* Cognitive mapping

46

Sources of Evidence Supporting Score Meaning

Source Examples

Evidence Based on Analyses of Hypothesized Relationships

Internal* Coefficient alpha, KR-20* Confirmatory factor analysis* Correlations among subscoresExternal* Correlations with criterion variables; convergent and discriminant analyses* Investigations of mean differences for relevant groups (e.g., treated/untreated; pre-treatment/ post-treatment; males/females, etc.)* Multi-trait, multi-method analyses

Evidence Based onTest Development and Administration Processes

* Item/task generation procedures* Bias/sensitivity reviews* Test administration and scoring procedures* Test security procedures

Conclusion

“In developing [the Standards], the organizations have put the requirement for evaluation of proposed interpretations and uses under the heading of validity. We can change that and put some or all of these issues under some other heading... but if we do so, we will have to reformulate much of the standard advice provided to test developers and test users.”

(Kane, 2009, p. 62)

Questions? Comments?

Thank you...

49