Intelligence testing: A matter of life and death?

Chad W. BuckendahlBuros Center for Testing

University of Nebraska, Lincoln

Buros Center for Testing• Oscar K. Buros, 1935; UNL, 1979• Improve the science and practice of testing• Buros Institute of Mental Measurements

(BIMM): commercially available tests– Mental Measurements Yearbook (MMY)– Test in Print (TIP)

• Buros Institute for Assessment Consultation and Outreach (BIACO): proprietary tests– Psychometric services for educational

licensure, certification, admissions, and employment testing

Case Background

• Defendant (Erick Vela) participated in a bank robbery in Norfolk, NE in September, 2002

• Five people were killed during the robbery

• One of four defendants convicted• Motion asserting mental retardation

Legal Background

• Atkins v. Virginia (2002)– Prohibited execution of persons with

mental retardation• Additional Nebraska statute

– “An intelligence quotient of 70 or below on a reliably administered intelligence quotient test shall be presumptive evidence of mental retardation.”

Diagnosing Mental Retardation

• Relies on multiple criteria• American Psychiatric Association and

American Association of Mental Retardation:– Sub-average intellectual functioning,– Significant limitations in adaptive skills

such as communications, self-care, and self-direction, and

– Manifest before age of 18

Defining primary criteria

• Sub-average intelligence:– IQ 70 or below (2 std. dev. below mean)– Considers measurement error

• Adaptive functioning:– Coping with common life demands– Standards of personal independence– Socio-cultural background– Community setting

Intelligence tests

• Tests administered by three different defense experts:– Wechsler Abbreviated Scale of

Intelligence (WASI): July 2003– Wechsler Adult Intelligence Scales (3rd

ed.) (WAIS-III): November 2003– Stanford-Binet (5th ed.) (SB5): July 2004

Adaptive Behavior Tests

• July 2003: none• November/December 2003: none• 2004-05: Vineland Adaptive Behavior

Scales – Defense: defendant’s sister– State: two friends of the defendant

Malingering tests• Administered when there are concerns

about motivation/effort in performance– July, 2003: none– November, 2003:

• Test of Memory Malingering (TOMM)• Victoria Symptom Validity Test (VSVT)• Validity Indicator Profile (VIP)• 21-item test

– July, 2004: none

Professional Standards

• Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999)– Validity– Reliability– Administration– Scoring– Score Interpretation

Validity Argument

• “Ultimately, the validity of an intended interpretation of test scores relies on all the available evidence relevant to the technical quality of a testing system. This includes . . .appropriate test administration and scoring. . .”(Standards, p. 17)

Administration and Scoring

• “The usefulness and interpretability of test scores require that a test be administered and scored according to the developer’s instructions. . . Without such standardization, the accuracy and comparability of score interpretations would be reduced.” (Standards, p. 61)

Scores

• WASI: 87 (Full Scale IQ)– 82 (Verbal IQ), 94 (Performance IQ)

• WAIS-III: 75 (FSIQ)– 75 (VIQ), 78 (PIQ)

• SB5: 66 (FSIQ)– 56 (VIQ), 79 (Nonverbal IQ)

Interpreting scale scores

• Full scale score relative to average (mean) score for population (i.e. 100)

• Full scale score relative to the statutory cut score (i.e. 70)

• Expected range of full scale and sub-scale scores given defendant’s observed scores

Comparison of subject's observed WASI scores with the general population

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

50 60 70 80 90 100 110 120 130 140 150IQ Score

Den

sity

Overall WASI distributionFull Scale IQVerbal IQPerformance IQ

Men

tal R

etar

datio

n

Comparison of subject's observed WASI FSIQ score with a score of 70

0.00

0.05

0.10

0.15

0.20

50 55 60 65 70 75 80 85 90 95 100FSIQ

Dens

ity

Men

tal R

etar

datio

n

Given an observed WASI FSIQ score of 87, one would have an approximately 0.0000002% chance of having a true score of 70 or lower.

WASI caution

“[The WASI] should not be used alone to make diagnosis or educational placement decisions. It is not meant to replace more comprehensive measures of intelligence, such as the WISC-III and the WAIS-III."

(Technical Manual, p.8)

Comparison of subject's observed WAIS-III scores with the general population

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

50 60 70 80 90 100 110 120 130 140 150IQ Score

Den

sity

Overall WAIS-IIIFull Scale IQVerbal IQPerformance IQ

Men

tal R

etar

datio

n

Comparison of subject's observed WAIS-III FSIQ score with a score of 70

0.00

0.05

0.10

0.15

0.20

50 55 60 65 70 75 80 85 90 95 100FSIQ

Dens

ity

Men

tal R

etar

datio

n

Given an observed WAIS-III FSIQ score of 75, one would have an approximately 1.7% chance of having a true score of 70 or lower.

“. . .a low score on the WAIS-III does not necessarily reflect a low level of intellectual functioning. . . other factors such as the following may be implicated: cultural and linguistic discrepancy from the test’s standardization sample, disabling distractibility or anxiety, severe psychopathology, deafness, poor motivation or inadequate persistence, and extremely oppositional behavior or very poor rapport with the examiner. Before diagnosing low intellectual functioning or mental retardation, the examiner must rule out these factors.” (emphasis added)

Administration and Scoring Manual (pp.7-8)

Interpreting WAIS-III Scores

Comparison of subject's observed SB5 scores with the general population

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

50 60 70 80 90 100 110 120 130 140 150IQ Score

Den

sity

Overall SB5Full Scale IQVerbal IQNon-Verbal IQ

Men

tal R

etar

datio

n

SB5 VIQ/NVIQ Differences

• The defendant had a…• Full scale IQ (FSIQ) score of 66• Verbal IQ (VIQ) score of 56• Nonverbal IQ (NVIQ) score of 79• 23 point difference between VIQ and NVIQ

• Only 1.9% of the standardization sample had a difference of 23 or more between VIQ and NVIQ

“… when significant differences exist between the NVIQ and the VIQ, examiners should be cautious about evaluating FSIQ as a summary of an individual’s general ability level. . .When the context of the assessment and examinee’s background is influenced by such factors as communication disorders, learning disabilities, autism, or non-English background, the NVIQ may be a better indicator of global cognitive potential.”(emphasis added)

Examiner’s Manual (p. 134)

Interpreting SB5 Scores

Comparison of subject's observed SB5 NVIQ score with a score of 70

0.00

0.05

0.10

0.15

0.20

50 55 60 65 70 75 80 85 90 95 100NVIQ

Dens

ity

Men

tal R

etar

datio

n

Given an observed SB5 NVIQ score of 79, one would have an approximately 0.1% chance of having a true score of 70 or lower.

Full Scale Scores Over Time

50

60

70

80

90

100

110

120

130

140

150

WASI, 07/200

3

WAIS-III, 1

1-12/200

3

SB5, 07

/2004

Full

Scal

e IQ

Verbal Scale Scores Over Time

50

60

70

80

90

100

110

120

130

140

150

WASI, 07/200

3

WAIS-III, 1

1-12/200

3

SB5, 07

/2004

Verb

al IQ

Nonverbal Scale Scores Over Time

50

60

70

80

90

100

110

120

130

140

150

WASI, 07/200

3

WAIS-III, 1

1-12/200

3

SB5, 07

/2004

Perf

orm

ance

/Non

verb

al IQ

TestWASI FSIQ <.00001 Verbal IQ 0.06 Performance IQ <0.00001WAIS-III FSIQ 1.7 Verbal IQ 2.7 Performance IQ 1.9SB5 FSIQ 95.9

Verbal IQ 97.0Nonverbal IQ 0.1

Chance of true score being 70 or lower (%)

Summary of Intelligence Tests

Adaptive Functioning Information

• Vineland Adaptive Behavior Scales• School records

– Course grades, remedial programs• Observations

– Family members, teachers, inmates– Employment record, corrections officials

Defense approach

• Strict interpretation of the statute (i.e. reliably administered intelligence test)

• Suggested that accuracy (validity) could be obtained without precision (reliability)

• Consider the standard error in determining the range of possible true IQ scores

• Suggested that observed scores across tests were different due to the Flynn effect (i.e. intergenerational influence on norm sample)

Judge’s Ruling

• Presumption of MR rejected• Validity of presumptive score:

– 3rd intelligence test administered by defense

– Probability of 66 on SB5 was remote– No malingering tests administered on it– Examiner did not follow interpretation

directions in test publisher’s manual

Judge’s Ruling (cont.)

• Sub-average intelligence– Considered standard error– WAIS-III score met the criterion

• Adaptive functioning– Preponderance of evidence did not support

limitations in this area• Age of onset

– NE’s statute does not specify age• Defense motion denied

Future Research Questions• Measurement

– Application of standard setting methodology for cut score– Validity of score interpretations given intended uses– Instrument development (e.g., content, norm sample)

• Legal– Implications for future cases/legislation– Court’s application of current validity theory

• Psychological– Criterion definition of mental retardation (rather than norm)– Interpretation of intelligence as unique from achievement– Stability of intelligence as a construct