the rorschach: facts, fictions, and...

Psychological Assessment2001, Vol. 13, No. 4, 452-471

Copyright 2001 by the American Psychological Association, Inc.1040-3590/01/S5.00 DOI: 10.1037//1040-3590.13.4.452

The Rorschach: Facts, Fictions, and Future

Donald J. ViglioneAlliant International University

Mark J. HilsenrothAdelphi University

A large body of empirical evidence supports the reliability, validity, and utility of the Rorschach. Thissame evidence reveals that the recent criticisms of the Rorschach are largely without merit. This articlesystematically addresses several significant Rorschach components: interrater and temporal consistencyreliability, normative data and diversity, methodological issues, specific applications in the evaluation ofthought disorder and suicide, meta-analyses, incremental validity, clinician judgment, patterns of use, andclinical utility. Strengths and weaknesses of the test are addressed, and research recommendations aremade. This information should give the reader both an appreciation for the substantial, but oftenoverlooked, research basis for the Rorschach and an appreciation of the challenges that lie ahead.

A large body of empirical evidence supports the reliability,validity, and clinical utility of the Rorschach, suggesting that therecent criticisms of the Rorschach are largely without merit. Weaddress 10 areas in which the Rorschach has recently been chal-lenged: (a) interrater reliability, (b) temporal consistency reliabil-ity, (c) normative data and diversity, (d) specific applications in theevaluation of thought disorder and suicide, (e) meta-analyses, (f)incremental validity, (g) clinician judgment, (h) implications ofclinical and forensic usage, (i) clinical utility, and (j) methodolog-ical issues relevant to validity. In each section, we summarize theempirical findings and their implications, explore the challenges,and recommend areas for future investigation to enhance theclinical utility of the Rorschach in personality assessment, so thatwe may increase our ability to understand our clients and tointervene effectively. This review will give the reader an appreci-ation for the substantial, but often overlooked and discounted,research basis for the reliability, validity, and utility of theRorschach.

Reliability

Interrater Reliability

Individual research studies published over the past 20 years inpeer-reviewed journals (Acklin, McDowell, & Verschell, 2000;McDowell & Acklin, 1996; Meyer, 1997a, 1997b; Viglione,1999), in Exner's volumes (Exner, 1993), and in the publication

Donald J. Viglione, California School of Psychology at Alliant Interna-tional University; Mark J. Hilsenroth, Derner Institute of Advanced Psy-chological Studies, Adelphi University.

We extend our appreciation to William Perry and Leonard Handler fortheir thoughtful comments on an earlier version of this manuscript. Thanksto our students for their help in collecting materials for this review: StevenAckerman, Matthew Baity, Nicole Dizon, Erin Eudell, Alice Laing, Kath-leen Montemagni, Joseph Obegi, Suzanne O'Brien, Maria Ortiz, and ToddPizitz.

Correspondence concerning this article should be addressed to Donald J.Viglione, California School of Psychology at Alliant International Univer-sity, 6160 Cornerstone Court, San Diego, California 92121.

process (Meyer et al., in press) provide conclusive empiricalevidence of strong interrater reliability for the great majority(95%) of Comprehensive System (CS) variables. An extensiveamount of data reveals that well-trained raters can score both highor low base-rate CS variables with good (K and intraclass corre-lation [ICC] > .60) to excellent (>.75-.80) reliability (Garb, 1998;Shrout & Fliess, 1979).1

Conclusion: Empirical Data Support Rorschach InterraterReliability

The great majority (95%) of the individual variables are codedwith good or excellent interrater reliability. Interrater reliability forform quality, individual cognitive special scores, form versus colordominance, shading subtypes, DOv and DOv/+, and some contentcategories could be enhanced. Only a few very low base-ratevariables (e.g., MQ none) occasionally produce poor reliabilitycoefficients, but they have not yet been analyzed with sufficientlylarge samples given their miniscule base rates. As in the past(Exner, 1974, 1995; Viglione, 1997), Rorschach researchers haverecognized the limitations of the test. Improvement in training forthe coding of these variables is the focus of a new reference bookcurrently being prepared for publication (Viglione, 2001).

Empirical Facts Regarding Temporal ConsistencyReliability

There is general agreement (Meyer, 1997a; Viglione, 1999) withWood and Lilienfeld's (1999) statement that "Exner and his col-leagues have reported test-retest reliability coefficients and thatthese coefficients are generally quite good" (p. 344). The greatmajority of the CS variables with such strong temporal consistencyreliability are central interpretive variables (Exner, 1991,1993). In

1 Our review of interrater reliability only addresses CS reliability. Manyother studies have demonstrated good to excellent reliability data fornon-CS variables such as the Mutuality of Autonomy (e.g., Fowler, Hilsen-roth, & Handler, 1996), Aggressive Content (e.g., Gacono & Meloy, 1994),and the Rorschach Oral Dependency scales (e.g., Fowler et al., 1996).

452

SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 453

other words, these variables are listed among the 68 nonredundantvariables on the lower portion, "below the line" in the CS struc-tural summary blank. In turn, the great majority of variableswithout test-retest coefficients (e.g., those identified by Garb,Wood, Nezworski, Grove, & Stejskal, 2001) are not includedamong these central, interpretive ratios, percentages, and deriva-tions. Thus, the variables supposedly without reliability coeffi-cients are not as central to interpretation. Of the 51 variablesidentified by Garb et al. (2001) as having test-retest coeffi-cients, 46 (90%) are central, interpretive ratios, percentages, andderivations. Consequently, the great majority of the most impor-tant variables have adequate data for temporal consistency reli-ability evaluation.

Garb et al. (2001) presented two tables addressing the issue ofCS test-retest reliability. Their Table 1 presents the central inter-pretive variables that have test-retest reliability coefficients.Among the 30 variables that Garb et al. identified (in their Table2) as not having test-retest coefficients, there are 3 variables(FC:CF + C, a:p, EB) that actually do have adequate data.FC:CF + C, a:p, EB are each ratios of two other variables, and allsix of the component scores are listed by Garb et al. in theirTable 1 as variables for which test-retest coefficients have beenreported. The text in a study by Exner, Armbruster, and Viglione(1978) clearly reveals that evidence of good reliability for EB,EA:ep (now called EA:es), and FC:CF + C cannot be attributed tochance. With respect to other variables that Garb et al. said lackedretest data, one, EBPer, is actually just a cutoff score of EB, twoof the listed scores do not exist [F-% and H + (Hd):(A) + (Ad)],and one variable is listed (blends), even though Exner (1999) hasreported on its retest stability. In addition, Garb et al. listed theAn + Xy score twice, listed the W variable twice, first as W andthen as W:M, and listed the variable M (in the W:M comparison),even though it is also listed in their Table 1 as one of the scoresthat has test-retest data. The variables CONTAM, PSV, and CP arelisted, but they have such incredibly low base rates that correla-tional, parametric statistics are unstable without huge samples.Among the 30 variables that Garb et al. (2001) alleged to lacktest-detest correlation coefficients, there are only 12 variables thatactually do not have test-retest data.

Table 1 is a more complete listing of the available test-retestcoefficients. These test-retest coefficients compare favorably tothe Wechsler scales, second version of the Minnesota MultiphasicPersonality Inventory (MMPI-2) and other tests. Table 2 presentsdata culled from many sources, comparing CS test-retest correla-tions with those from these other scales. Table 2 presents the recentmeta-analyses that address this issue, an exhaustive selection ofrecent MMPI and MMPI-2 data, and data on two other represen-tative tests, one a performance test and the second a commonlyused research instrument.

Memory effects cannot account for strong temporal consistencyreliability findings. With respect to retest intervals of 6 months ormore, psychometric texts (e.g., Anastasi & Urbina, 1996) mini-mize the possibility that memory effects act as confounds. It iscertainly not an issue in 3-year retest intervals (Exner et al., 1978).In addition, Haller and Exner (1985) addressed the memory issueempirically. They instructed experimental respondents to givedifferent answers on the second testing, but they gave no suchinstructions to the control groups. The experimental group pro-duced adequate temporal consistency reliability coefficients (mean

correlation = .66; Mdn = .71), which were almost identical tothose of the control group (mean correlation = .72; Mdn = .74).Nonetheless, a review of the responses revealed that experimentalparticipants gave many more new responses in the second testingthan did control participants. Thus, even though the content of theresponses changed, the formal scores for the central interpretivevariables largely remained the same.

The six actuarial indexes in the CS (i.e., Schizophrenia [SCZI],depression [DEPI], suicide constellation [S-CON], coping deficit[GDI], hypervigilance [HVI], and obsessive style [OBS]) lackdefinitive temporal consistency data, but available findings supporttheir stability. The actuarial indexes are derived from 58 vari-ables, 47 of which possess or adequate test-retest data in theresearch reports. Six of the 11 scores without test-retest data (FQo,FQ-- FQu, Level 2 special scores, and FABCOM2) are subcom-ponents of other variables (X—%, X+%, Ego Impairment Index[EII], WSUM6, SUM6, INC + FAB) with good to excellent test-retest data.

Garb et al. (2001) cited several studies that seemed to demon-strate less acceptable temporal consistency reliability for the CS,but they did not report their serious methodological flaws. Theseflawed studies include research with a nonstandard form of ad-ministration (Schwartz, Mebane, & Malony, 1990); a dissertationinvolving only 17 older adults with an unspecified test-retestinterval (Erstad, 1996); and a sample of schizophrenic patientswith various illness courses, medications, hospitalizations, psycho-therapies, and test-retest intervals ranging from 1 to 18 years(averaging 6.4 years; Adair & Wagner, 1992).

Conclusion and Future Research

The empirical data support the conclusion that the great majorityof CS central interpretive ratios, percentages, and derivations pos-sess impressive temporal consistency. As demonstrated in Table 2,the test-retest coefficients for CS Rorschach variables comparequite favorably with those for other tests. For the limited numberof variables without published retest coefficients, most are ofsecondary interpretive significance. Typically, they enter into cal-culations to produce the central interpretive variables, and thelatter have demonstrated adequate or better temporal consistency.The CS has more than adequate temporal consistency reliability inall respects, especially in the context of comparing Rorschach datato other established methods of assessment.

Nonetheless, more test-detest reliability research should be un-dertaken. The Rorschach and the CS, like many other psycholog-ical assessment measures (e.g., MMPI/MMPI-2 and WechslerAdult Intelligence Scale—Revised [WAIS-R/WAIS-III]), con-tinue to evolve; therefore, temporal consistency informationshould be updated and include variables for which no data areavailable. There were only 22 ratios, percentages, and derivationsin the CS structural summary and approximately a dozen othervariables central to interpretation (Exner, Weiner, & Schuyler,1976) at the time that the original temporal consistency data wereprocessed (Exner, Armbruster, & Viglione, 1978). Future researchcould utilize short test-retest intervals and ensure minimal changein psychiatric, environmental, and target constructs. The sampleshould include both nonpatients and a variety of patients to avoidrestrictions of range on actuarial indices and pathology indicators.To analyze data with moderately low base rates, a large sample

Table 1Test-Retest Correlation Coefficients for Comprehensive System Structural Summary Variables Among Adults

Group8

Variables" 1 2 3 4

NRetest periodParticipants

Complexity, style, controlsRLambdaEff* (M:WSumQ

MWSumC

EAa

EMmCVTYeb (FM + m:Sum Sh)

FM + mSumSh

esD-ScoreAdj esd

AffectFCe

CF + Cf

C + CnAfr.SBlend?

50a 1 year

Adult nonpatients

.86

.78

.84

.82

.83

.77

.26

.73

.87

.91

.31

.64

.91

.82

.86

.81

.56

.82

.74

100z 3 years

Adult nonpatients

.79

.82

.87

.86

.85

.72

.39

.67

.81

.87

.23

.69

.66

.72

.83

.86

.79

.51

.90

353 weeks

Adult depressed inpatients

.84

.76

.83

.83

.84

.72

.34

.67

.89

.96

.41

.71

.59

.88

.85

.92

.83

.59

.85

.71

253 or 4 days

Adult nonpatients

.77

.82

.75

.68

.71

.28

.84

.77

.82

.86

.69

.74

.69

.71

.74

.57

572 years

Adolescent nonpatients age 14 to 16

.80

.78

.77

.69

.71

.81

.06

.69

.92

.91

.13

.71

.73

.72

.76

.18

Ideation<f .83 .86 .87 .71 .83p" .72 .75 .85 .51 .782AB + Art + Ay (Intell.) .84 .88Sum6 special scores .81 .88 .81 .84f .68f

WSum6 .86 .86

MediationP .83 .73 .81 .76 .82X+% .86 .80 .87 .82 .85X-% .92 .80 .88Xu% .85 .89

Information processing7f .85 .83 .89 .797d .70

InterpersonalCOP .81 .88AG .82 .86Isolate/R .84 .83

Self-perception3r + (2)/R .89 .87 .90 .74 .73Fr + rF .82 .89FD .88 .90 .76MOR .71 .83 .87

" Citations for the groups are as follows: Group 1 = Exner, 1993; Exner, 1999. Group 2 = Exner, Armbruster, & Viglione, 1978; Exner, 1993; Exner &Weiner, 1995. Group 3 = Exner, 1993, 1999; Exner & Weiner, 1995. Group 4 = Mailer & Exner, 1985. Group 5 = Exner, Thomas, & Mason, 1985.b For those variables listed as missing by Garb et al. (2001), and for which we could find no published retest coefficients, we asked Exner to provide thecoefficients. Garb et al. (2001) did not list M—, DQ+, DQv, and Level 2 cognitive special scores as missing. Accordingly, we did not process coefficientsfor these variables and do not present mem in the table. CP and Mnone are listed as missing by Garb et al. (2001). They have extremely low base rates.Only 5 of the approximately 13,392 responses in the nonpatient reference group of 600 adults (Exner, 2001) were assigned a CP and only 4 were assigneda Mnone. Accordingly, a huge sample is needed to stabilize the retest coefficients for these two variables and values were not requested for them. Test-retestcoefficients are still needed for Ma, Mp, Level 2 cognitive special scores, W. D, Dd, DQ+, DQv, Fd, Pure H, (H) + Hd + (Hd), An + X>, and the newcomprehensive system variable (Exner, 2001).c EBPer is merely a cutoff score of EB. It is listed as missing by Garb et al. (2001).d Adj D is merely a transformation of EA—Adj es, both of which have coefficients.c Listed by Garb et al. (2001) as lacking test-retest coefficients.f At the time of these publications (Haller & Exner, 1985; Exner, Thomas, & Mason, 1985), there were only five cognitive special scores.


Table 2A Comparison of Test-Retest Correlations Among FrequentlyUsed and Highly Regarded Tests

Tests variables

Correlation

Retest interval M Mdn

Multiple studiesMeta-analysis—Self-report, observer.

and projectivea

Meta-analysis — 8 self-report testsb

Meta-analysis — MMPfMMPI— 7 studies0

Single studiesMMPI"MMPI-2

California Verbal Learning Test

Attributional Style Questionnaire1

Rorschach

1 year5 years1 year5 years5 years1 year

1 day to 2 years1 week*4 monthsf

5 years8

1.3 years"1 year1

5 weeks'3 years*3 weeks1

1 year1

3 weeks'1 year1

1 year"1

3 to 4 days3 to 4 days

.55

.52

.78

.58

.57

.57

.68

.80

.73

.72

.47

.53

.63

.79

.80

.76

.82

.82

.75

.71"

.66°

———

——.58

.68

.83

.74

.73

.43

.54

.64

.80

.85

.82

.85

.83

.82

.74

.71

Note. Dashes indicate measure was not applicable. MMPI = MinnesotaMultiphasic Personality Inventory.1 Roberts & DelVecchio (2000). b Schuerger, Zarrella, & Holtz,(1989). °Mauger (1972) as cited in Dahlstrom, Welsh, & Dahlstrom(1975); Stone (1965) as cited in Dahlstrom et al. (1975); Sines, Silver, &Lucero (1961) as cited in Dahlstrom et al. (1975); Ryan, Dunn, & Paolo(1995); Milott, Lira, & Miller (1977). "Hunsley, Hanson, & Parker(1988). e Butcher, Dahlstrom, Graham, et al. (1989). 'Putnam, Kurtz,& Houts (1996). g Spiro, Butcher, Levenson, Aldwin, & Bosse (1993) ascited in Putnam et al. (19%). "Paolo, Troster, & Ryan (1997). 'Deliset al. (1996) as cited in Paolo et al. j Peterson et al. (1982). kExner,Armbruster, & Viglione (1978). 'Exner (1999). m Exner (1993)." Mailer & Exner (1985), depressed inpatients. ° Haller & Exner (1985),depressed inpatients instructed to give different answer to evaluate "mem-ory effects."

(N = 80 or more) should be collected. Interpretive temporalconsistency based on established interpretive cutoffs should beincluded. Systematically assigning multiple raters to recederecords would allow one to discriminate the effects of temporalinconsistency from interrater disagreement in accounting forchanges over time.

Normative Data and Diversity Issues

The detailed presentation of large reference samples from non-patients, various patient groups, and children from ages 5 to 16 hascontributed to a resurgence of the Rorschach over the past 20 years(American Psychological Association, [APA] 1998; Exner, 1974,1993; Exner et al., 2001; Wood & Lilienfeld, 1999). To limit ourdiscussion of normative data, we focus here on the adult, nonpa-tient CS reference sample, consisting of 600 adults, which is arelatively large normative sample (Exner et al., 2001). It is strat-

ified for geographic distribution and partially stratified for socio-economic level. The first of these nonpatient participants wascollected before 1975, followed by various additions and revisionsthrough 1991 (Exner, 1993). All respondents were volunteers, withthe majority coming from places of employment. The genderdistribution is balanced, but the sample is rather young; 73% of theparticipants were under 36 years of age. Nonetheless, there aresome differences between this adult group and the 15- and 16-year-old sample (Exner et al., 2001). The adult group is relativelywell-educated; the median grade level is over 13 years, and only5% had less than 12 years of schooling. Eighteen percent of therespondents were non-Caucasian, which is consistent with othernormative samples, although this percentage could be increased.For example, the minority percentage for the WAIS-III is 21%(The Psychological Corporation, 1997) and for the MMPI-2 it is19% (Hathaway et al., 1989).

Rorschach researchers (Brinderson, 1995; Kates, 1994; Shaffer,Erdberg, & Haroian, 1999; Viglione, Gaudiana, & Gowri, 1997;Weiner, 200 Ib) have questioned the continued suitability of theseCS norms. For reasons similar to the emergence of concerns aboutthe aging of MMPI norms (e.g., Colligan, Osborne, & Offord,1984), one might contend that social and cultural shifts, cohortdifferences, and the gradual evolution of the test justify collectionof a new sample.

Research Data as Approximations of Nonpatient Sampling

To approximate a contemporary nonpatient reference sample,we reviewed the literature for available adult control and nonpa-tient groups. Shaffer et al. (1999) published the most comprehen-sive and notable effort to produce normative approximation data.To construct Table 3, we selected central interpretive variables thatappeared to deviate from CS normative expectations in the Shafferet al. study. We then supplemented the Shaffer et al. findings withdata from five other published samples known to us and onepreviously presented sample (Viglione et al., 1997). These dataallowed us to address some issues regarding deviation from CSreference sample expectations.

R. The data in Table 3 suggest that R is quite variable.Whether or not such variability presents a problem for the Ror-schach has been debated for many years. The distributional pa-rameters (mean is greater than median, a large standard deviation,and a minimum value of 14 with no predetermined maximumvalue) suggest that a great deal of variability is due to a smallproportion of records with excessively high R. From a clinicalutility perspective, long records consume administration and cod-ing time and increase cost without necessarily increasing interpre-tive yield.

Lambda. The data for Lambda are incomplete, but suggest thatmedian values are close to CS reference values. The Shaffer et al.(1999) study provides the only sample that seems to deviate fromthe expected distribution. Nonetheless, on the basis of standarddeviations, within each sample there are records that are muchmore form dominant than would be expected. The occasional highLambda record does not pose much of a problem for routineinterpretation. On the other hand, low /f-high Lambda recordsmay lack negative predictive power and may reduce interpretiveyields and cost-benefit ratios.

456 VIGLIONE AND HILSENROTH

Table 3Nonpatient Adults' Data for Some Rorschach Variables

Lambda" x-w WSVM6

N M Mdn SD M Mdn SD M SD M SD M Mdn SD

20"20b

35C

25"123e

35f

83s

24517.626.0

20.820.622.8

245

182121

933.77.3

757.77.3

0750.680.831.01.2

0.58

061

075

0.45

0510.580.570.801 72

0.46

0550.520.520.570.51

0.55

0150.140.130.180 15

0.14

0200.170.21

0210.160.17

0 15

0.110.130.17

909.72.8

662.17.9

38

400.05.0

12311.43.8

802.99.3

Summary means across studies

Weighted 21.9 21.5 7.3 0.92 0.63 1.01 0.53 0.15 0.19 0.14 6.4 4.4 7.8Not weighted 22.1 22.2 7.1 0.84 0.60 0.77 0.54 0.15 0.19 0.14 6.4 4.3 7.9

21.9 22

Exner et al. (2001) reference sample

4.4 0.60 0.53 0.31 0.77 0.09 0.07 0.05 4.5 5.0 1.8

"Netter & Viglione (1994), volunteers—no history of psychological hospitalizations or medication, denysubstance abuse. bFrueh & Kinder (1994), White, male college students. cGreenwald (1991), collegestudents. dSloan, Arsenault, Hilsenroth, Handler, & Harvill (1996), White, male, military reservists. "Shaf-fer, Erdberg, & Haroian (1999). f Bums & Viglione (19%), female volunteers with superior interpersonalrelationships. g Viglione, Gaudiana, & Gowri (1997), nonpatient volunteers, without history of psychiatrictreatment, deny substance abuse. h Because of significant problems with skew for this ratio as the denominator,Pure F, approaches zero, means and standard deviations may be misleading. The large standard deviations withmeans greater than medians suggest that skew is present and that the means are artificially elevated by outliersor skewed high values. ' Distributions are quite normal for X+% and X—%, so that only means and standarddeviations are presented.

Form quality. X+% and X—% results suggest that FQu andFQ— are more numerous than anticipated, but they are consistentwith some previous observations (Viglione, 1989). Interpretiveranges for X+%, as a measure of overall conventionality, andX— %, as a measure of perceptual distortion, need to be modified.The data in Table 3 may underestimate X+% and overestimateX—% because one cannot be sure that pathological respondentswere excluded from the samples. Nevertheless, the results dosuggest that a more appropriate cutoff for the "unconventional"interpretation of X+ % may be approximately X+ % < .50 and thatthe cutoff for the "distortion" interpretation of X—% may be closerto X-% > .25. The data suggest that the cutoffs for childrenshould also be more liberal (Hamel, Shaffer, & Erdberg, 2000;Kates, 1994). With improvements in coding FQ, as suggested inthe interrater reliability section in this article, these interpretivebands might tighten. It is also important to recognize that FQ hassome of the best validity data of any coding CS variable (Dawes,1999; Viglione, 1999). Cases with elevations in the SCZI producedsolely by form quality should be closely examined to rule outpossible false-positive findings.2

Exner (2000) has made some changes in the form qualitypercentages to address these issues. However, if it is true that 20%of the responses in nonpatient records are distorted (i.e., FQ—),then we have a problem with the operational definition for theconstruct of perceptual distortion. One could justify 5% or 10% ofthe responses being distorted, but it does not make sense for us tocall 20% of Rorschach responses among nonpatients distorted. Wemay need to narrow the operational definition of FQ—, so that thiscode truly represents distorted perception as outlined by Dawes(1999).

WSUM6. For cognitive special scores in the form of WSUM6,the large standard deviations suggest that a small proportion ofindividuals provide many of the elevated values. Given that theserespondents are not patients (though it is possible that some hadbeen prior to testing), one cannot be sure whether these results areassociated with poor sampling, idiosyncratic responses, idiosyn-cratic coding, or overly long, elaborated, and complex records. Onthe other hand, central tendencies are consistent with the CS data.The standard deviations for WSUM6, as well as R and Lambda,suggest that these samples contain more exceptional or extremecases than does the CS reference group.

Cultural and International Implications

These normative approximation data also have implications forcultural and international issues. Research data on the Rorschachand diversity issues support the use of this measure with diversepopulations (Viglione, 1999). Large-scale cultural diversity andinternational projects have been under way for some time, andexpertise among the project managers around the world is improv-ing (Erdberg & Schaffer, 1999). These data have the potential ofteaching us a great deal about cultural and geographic issues. Interms of thinking about cultural differences and expectations from

2 The SCZI is being replaced by a dimensional index of thought disor-der, the Perceptual Thought Index (Exner, 2001). Such an index moreaccurately reflects the available data (Hilsenroth, Fowler, & Padawer,1998; Kleiger, 1999) and a modem understanding of thought disorder asexisting on a continuum from optimal to most disordered within and acrossdiagnoses (Viglione, 1999).


a normative perspective, the data emerging from these studies aremore consistent with the normative approximation data given inTable 3 than they are with the CS reference sample.

Moreover, emerging childhood reference group data fromUnited States samples (Hamel et al., 2000; Kates, 1994) andinternational samples (Erdberg & Schaffer, 1999) are consistentwith a normative approximation sample of African Americanchildren that was collected about 20 years ago (Krall et al., 1983).Garb et al. (2001) cited the Krall et al. study in their challenge ofthe diversity applications of the Rorschach. However, this is quitepuzzling because a close examination of the data indicates that theonly problematic difference between Krall et al.'s findings and thenorms available at that time (Exner & Weiner, 1982) is for a formquality score (i.e., a lower F+%), which is no longer being used inthe CS (Exner, 2000). Krall et al. reported a range for F+% of .48to .69, which is consistent with the normative approximationresults presented in Table 3 and the emerging international data.Also, differences in popular responses, for example, are as ex-pected given cultural variations (Mattlar, Carlsson, & Forsander,1993). Thus, supposed problematic cultural differences evaporatewhen one considers the emerging international data. This conclu-sion is supported by the very limited differences between nonpa-tient African American and Caucasian American adults matchedon important demographic variables (Presley, Smith, Hilsenroth, &Exner, in press). Finally, Meyer (in press) extended these findingsby conducting a series of analyses to explore potential ethnic biasin Rorschach CS variables with a patient sample. After matchingon several salient demographic variables, ethnicity revealed nosignificant findings, and principal-components analyses revealedno evidence of ethnic bias in the Rorschach's internal structure.

A New Normative Sample?

The Shaffer et al. (1999) and other normative approximationdata are relatively consistent with CS reference group data (Exneret al., 1995). Adjusting one's expectations for form quality, asoutlined above, should eliminate the possibility of overpathologiz-ing adults.

An alternative point of view about these reference data withnonpatients is also emerging (Meyer, 2001; Weiner, 2001b). Thispoint of view suggests that there should be a careful considerationof the guidelines used for collecting this Rorschach reference data.Figure! in this understanding of differences is the recognition thatExner's (1993) reference sample consists of socially-vocationallyfunctioning nonpatients. In addition, it is equally important to notethat CS data collected from both United States and worldwidesamples are somewhat more healthy than Exner's reference sam-ples of people who are starting outpatient psychotherapy or dif-ferent patient groups. The interested reader is directed to Weiner(2001b), who proposed guidelines for research when obtainingreference samples from patients and nonpatients. Meyer (2001)also provided an extensive and in depth review of possible factorsaffecting normative shifts in Rorschach data. Both of these authorspresented rational and empirical data to support their position thatCS norms do not overpathologize potential examinees.

Nonetheless, collecting new reference samples, with increasedattention to cultural and other diversity issues, should improvereliability and validity, and enhance utility. Like the MMPI, the CSand Rorschach have evolved since they were introduced (Exner,

1974). The findings with R, and the possibility of related variabil-ity among the other indices, suggest that constraining R throughstatistical comparisons or procedurally through minor administra-tive modifications deserves some consideration.

Meta-Analyses

Do meta-analytic results support the validity of the Rorschach ingeneral terms? There are at least six recent and original meta-analyses that address Rorschach validity (Atkinson, Quarrington,Alp, & Cyr, 1986; Bornstein, 1996,1999; Hiller, Rosenthal, Born-stein, & Berry, 1999; Meyer & Handler, 1997; Parker, Hanson, &Hunsley, 1988). All concluded with empirically based statements,supporting the validity of the Rorschach. Weiner (200la) summa-rized these empirical data and offered them as convincing supportof the overall validity of the Rorschach. Opposing these fivemeta-analyses is one re-analysis of the Parker et al. data (1988) byGarb, Florio, and Grove (1998). The research in this meta-analyseswas published from 1954 through 1982 and contains only 15Rorschach studies (compared with 75 utilizing the MMPI), only 5of which utilized the CS. As suggested in the original authors'(Parker, Hunsley, & Hanson, 1999) reply to the re-analysis of theirdata, the relevance of the Garb et al. (1998) re-analysis is highlyquestionable and adds little to our contemporary understanding ofthese two clinical instruments. The Rorschach research and tech-niques have advanced quite a bit over the last 20 years. Likewise,the MMPI-2 was not available to be evaluated in either of thesetwo meta-analytic studies.

Analysis of moderator variables in the Hiller et al. (1999)meta-analysis in this Special Series revealed that the MMPI hadlarger validity coefficients than the Rorschach for self-report cri-teria and psychiatric diagnosis "whereas the Rorschach had highervalidity coefficients in studies using objective outcomes as crite-rion variables" (Hiller et al., 1999, p. 291). This is consistent withViglione's (1999) finding that the Rorschach variables were validand useful when one emphasizes ecologically valid behavioralcriteria, rather than self-report criteria. Thus, as the criterion im-proves, so do the Rorschach validity coefficients. In contrast, theMMPI does best with criteria that share self-report method vari-ance. Hiller et al. speculated that impression management playsless of a part in the Rorschach than in self-report testing. "In theseinstances, it may be that the Rorschach is more objective thanso-called objective instruments" (p. 291). Although Hiller,Rosenthal, and their colleagues' methods, data, and conclusionshave been critiqued, the support for the Rorschach from theirwork, as well as from all the other original meta-analyses, are quiteclear and accurate. As a whole, these meta-analyses reveal that theRorschach produces valid data to measure many objective, behav-ioral criteria.

Incremental Validity

Research data from a variety of sources reveals considerableemerging empirical support for the incremental validity of theRorschach. This research is not exhaustive, but it is clear thatRorschach variables do possess incremental validity over otherwell-known tests and collateral information. As such, the evidenceindicates that the Rorschach adds meaningful data to the assess-


ment process. Accordingly, negative characterizations, such as"The frequent claim that the Rorschach adds meaningful assess-ment information to other data has not been supported in anypublished study" (Hunsley & Bailey, 1999, p. 271), are falsebecause they ignore empirical findings.

Empirical Data That Address Incremental Validity

Research presented in Weiner's (2001a) and Viglione's (1999)reviews of the literature contain empirical data consistent with theconclusion that Rorschach variables possess incremental validityover other tests, demographic data, and other information (see

Table 4). These data were not sufficiently recognized by those whoadopted a critical view of the Rorschach in this Special Series. Onthe other hand, consistent with Weiner, Garb (1999) did confirmincremental validity support to the EII, the Rorschach Oral De-pendency scale, and the Rorschach Prognostic Rating scale (PRS).

Meyer (2000) has addressed incremental validity issues in twometa-analyses. Within six studies, using both the Rorschach PRSand the MMPI Ego Strength scale (Es), the Rorschach PRS dem-onstrated considerable incremental validity for predicting treat-ment outcome. The uncorrected and corrected effect sizes werer = .32 and r = .48 (N = 229), respectively. The uncorrected andcorrected effect sizes, after excluding the outlier results from one

Table 4Studies Included in Viglione (1999) With Findings Consistent With Rorschach Incremental Validity

Author and year Findings

Archer & Gordon (1988)

Archer & Krishnamurthy (1997)

Bornstein et al. (1997)

Cooper et al. (1991)

Hilsenroth et al. (1995)

Holzman et al. (1974)

O'Connell et al. (1989)

Perry & Braff (1994)

Perry & Viglione (1991)

Russ (1980)

Russ (1981)

Shapiro et al. (1990)

Skelton et al. (1995)

Optimal overall correct classification (OCC hit rate) of individual inpatients with schizophrenia forRorschach Schizophrenia (SCZI = 5) = .80. Optimal OCC for Minnesota Multiphasic PersonalityInventory (MMPI) Sc scale (Sc s 75) = .76. Utilizing traditional cutoff scores, OCC rates were asfollows: SCZI > 4, OCC = .69; Sc > 65, OCC = .48; and Sc > 70, OCC = .60.

In classification of depression in adolescents with a stepwise discriminant function analysis, Rorschachvariables Vista and Afr added R2 = .05 beyond combined R2 = .14 for MMPI-A scales A-DEP and Ma.These four variables had highest positive predictive power over any single variable or combination ofvariables.

Rorschach Dependency scores significantly correlated with both number of significant interpersonal events(r = .84) and impact ratings of these events (r = —.38, p < .01). Self-report measure of dependency wasnot significantly related to either (r = —.11 and r = .16, respectively).

The Rorschach Defense scales provided unique prediction of outcome GAP (Global Assessment ofFunctioning, Health-Sickness) ratings in regression equations beyond initial GAP and borderlinepersonality self-report scale.

Rorschach variables were able to significantly differentiate (p = .008) those patients prematurelyterminating from psychotherapy vs. those continuing in treatment, whereas the MMPI-2 was unable to doso (p = .56). Specifically Rorschach scores from the interpersonal-relational cluster had a mean effectsize (ES) of .57, while the MMPI-2 content scale Negative Treatment Indicators had an ES of -.14.

The classification of a recent schizophrenia diagnosis (hospitalized less than 6 months) and deviant eyetracking was greater (65% accuracy) utilizing Rorschach data alone than a clinical team diagnosis (58%).

Rorschach data (Thought Disorder Index [TDI]) predicted the development of psychotic and psychotic-likesymptoms over a 2-3 year period in a sample of Axis II and affective disorder patients over and aboveinformation from clinical interview on lifetime prevalence of psychotic and psychotic-like symptoms(combined R2 = .21, TDI-beta = .32, p < .03), schizotypal symptoms (combined R2 = .42, TDI-beta =.32, p < .009), or schizotypal and borderline symptoms (combined R2 = .51, TDI-beta = .31, p < .006).Initial TDI scores also demonstrated clinical utility in prediction of psychotic and psychotic-likesymptoms at follow-up.

Human Experience variable component of the Ego Impairment Index (EH) significantly related toneuropsychological markers of schizophrenia (r = -.42, p < .01; r = -.37, p < .025; r = -.35, p <.025), whereas thought disorder scales based on clinical interview (Schedule for Positive Symptoms andSchedule for Negative Symptoms) were not (p > .05).

Rorschach EH predicted outcome Beck Depression Inventory (BDI; p < .0002) and Carroll Rating Scale forDepression (p < .01) scores in depressed patients treated with tricyclic antidepressants beyond varianceaccounted for by gender and baseline scores on BDI and EH. Other demographic variables were alsoconsidered but did not affect outcome.

Rorschach measures of adaptive regression (AR) and defensive effectiveness (DE) were significantly relatedto academic achievement, independent of IQ (AR: r = .45, p < .01; DE: r = .40, p < .01, respectively).

Rorschach measure of AR was significantly related to reading and overall academic achievementindependent of IQ (r = .51, p < .001, and r = .47, p < .001). Index AR scores were significantlypredictive of reading achievement 1 year later (r = .29, p < .05).

Rorschach Depression Index significantly differentiated sexually abused African American girls fromcontrols (p < .005). Children's Depression Inventory scores for sample were not significantly differentfrom controls (p > .05), consistent with incremental validity of the Rorschach relative to self-reporteddepression. The groups did differ on the Intemalization scale of the Child Behavior Checklist (p <.0001).

The dependent variable was a ratio of Rorschach TDI over a TDI derived from the Wechsler. This ratiowas 2.46 times higher in a group of 25 identity-disordered adolescents than it was among 35 conduct-disordered and oppositional-defiant adolescents (p < .01).


study (Fiske, Cartwright, & Kirtner, 1964), were r = .40 and r =.59 (N = 187), respectively. Effect sizes for the Es were essentiallynil (range, -.03 to 03). Clearly, the Rorschach PRS demonstratesincremental validity over the MMPI Es scale. Nine studies (N =358) included both the Rorschach PRS and an intelligence test.Across these studies, the Rorschach PRS demonstrated consider-able incremental validity over and above intelligence in the pre-diction of future outcome. After correcting for methodologicalartifacts, the unique contribution for the PRS was r = .48; andwithout such corrections, r = .36. Blatt's (Blatt et al., 1994)findings also demonstrate incremental validity of Rorschach scalesover IQ in the prediction of treatment outcome, as do a number ofstudies summarized in Viglione's (1999) synthesis of the literature(see Table 4).

Blais, Hilsenroth, Castlebury, Fowler, and Baity (2001) ex-plored the incremental validity of the Rorschach in discriminatingDiagnostic and Statistical Manual of Mental Disorders (4th ed.;DSM-IV; American Psychiatric Association, 1994) personalitydisorders (PD). In a re-analysis of existing data, select Rorschachvariables and the MMPI-2, PD scales were used to predict chartreview ratings of DSM-IV antisocial, borderline, histrionic, andnarcissistic personality disorders in a sample of treatment-seekingoutpatients. Hierarchical regression analyses showed that bothMMPI-2 and Rorschach data added incrementally to the predictionof DSM-IV borderline and narcissistic PD criteria. The findingswere less clear for the incremental value of Rorschach andMMPI-2 data in predicting the total number of DSM-IV histrionicPD and antisocial PD criteria. Rorschach data were the best pre-dictors of DSM-IV histrionic criteria whereas MMPI-2 data werebest at predicting the DSM-IV antisocial criteria.

Dawes's (1999) Results Actually Support the Criterionand Incremental Validity of the Ell and Rorschach

Earlier in this Rorschach Special Series, Dawes (1999) "pre-sent[ed] two methods for assessing the incremental validity of aparticular Rorschach variable" (p. 297), namely the EH, with theexpressed purpose of demonstrating how incremental validity ofthe Rorschach might be investigated. He investigated two data setsfrom: (a) Meyer and Resnick (1996), which includes the Ror-schach, MMPI, and impairment as approximated by severity de-fined by diagnosis (DxMax); and (b) Perry, Moore, and Braff(1995), which contains the Rorschach, measures of social compe-tence and thought disorder, and another psychiatric rating scale.His conclusions are quite critical of the Rorschach but do not fitthe data he presents. The analytic strategies he used reduced theprobability of finding support for the Rorschach and EII and biasedthe analyses against the Rorschach. Therefore, we take issue withhis methods and conclusions in this article.

Reducing the power of the analyses. To re-evaluate these datasets, Dawes (a) entered the number of Rorschach responses (R) andhis new form quality measure, XQUAL, before the EII in thehierarchical regression; and (b) used a unit weighting scheme inaddition to multiple regression. The argument for using the numberof responses and his new measure, XQUAL, is that these Ror-schach variables are easier to score and involve less projection bythe clinician.3 There is no established body of research that sup-ports the number of responses to predict impairment; therefore,there is no a priori reason to justify it as a predictor of impairment

in a regression analysis. Also, data from the study do not supportincluding R as a predictor The number of responses is not signif-icantly correlated with the criteria (Meyer's DxMax, r = .13, nsand in the wrong direction; and Perry's Social Competence Index,r = .16, ns).

Similarly, Dawes's decision to create a new variable, XQUALand to require the EH to predict beyond XQUAL in a regressionequation cannot be justified. The EH contains form quality (FQ—)and R among its subcomponents. XQUAL and R are, thus, struc-turally redundant with EII and produce multicollinearity problemswithin the regression, reducing the power of the analyses. Amongthe arguments for the unit weighting system is that a clinician'sintuition does not reliably "capture optimal weighting systems" (p.299). Its effect in the analysis is to reduce power once again.

Garb et al. (2001) argued that Dawes's (1999) and Meyer andResnick's (1996) use of the severity of clinical diagnosis as acriterion is only a gross approximation of impairment or psycho-logical disturbance. This is true, as it is for the Social CompetenceIndex criterion in the Perry, Moore, and Braff (1995) data set.These measures do not share a great deal of variance with impair-ment, the intended target construct of the EII. This limited overlapbetween measure and target construct stacks the cards against theRorschach by limiting maximal effect sizes and the power of theanalyses. Small effect sizes are to be expected, even in the bestcircumstances, with such a global criterion. Under such circum-stances, the maximum multiple Rs of .49 and .35, reported byDawes, are quite impressive.

Dawes also introduced DxSum, the sum of the impairmentscores for up to three diagnoses, as a criterion measure.4 It wasused without precedent, supporting data, or a straightforward ra-tionale. The DxSum makes the sum of mild diagnoses much moreimpaired than the most virulent but pure schizophrenia or bipolardisorder with psychotic features and adds systematic error toDxMax. We obtained the Meyer and Resnick (1996) data set (N =187) and found that 36 individuals in the data set had a diagnosiswith moderate severity, namely a DxMax score of 3 or less, but aDxSum of 6 or more. For these 36 patients, even though they hadbeen assigned several diagnoses of mild to moderate severity, theirDxSum score made them appear more impaired than individualswith a single diagnosis of schizophrenia (N = 16). In addition, thedata do not support Dawes's argument: All correlations in Dawes'sTable 1 between meaningful predictors (the MMPI mean eleva-tion, the MMPI Goldberg index, XQUAL and EII) are higher forDxMax than for Dawes's DxSum. Clearly, the DxSum variableadds unacceptable error to the criterion and only functions to biasthe results against the Rorschach.

3 One may presume that some researchers and clinicians might disagreewith the statement that form quality distortions (FQ-) "involve neitherempirical research to establish their validity nor training in the Rorschachto score" (Dawes, 1999, p. 297). However, it appears that Dawes acceptsthat certain FQ— responses can be seen as clear, in vivo problematicbehaviors, consistent with understanding the Rorschach as behavioralsample of cognitive problem-solving behaviors (Viglione, 1999).

4 Dawes (1999, p. 298) stated that "the authors considered the averagerating for up to three diagnoses and the sum of the ratings for up to threediagnoses." Meyer (personal communication, March 6, 2000) calculatedthe average and sum of diagnoses, but he did not recall considering themto be valid criterion variables and did not include them in his analyses.


With respect to the Perry et al. data set, the only relevant statisticis the univariate correlation relating the EII to the Social Compe-tence Index (r= .31, p < .05). Also, on the basis of the additionalmeasures included in the Perry data set, the EII was correlated withthe Schedule for Positive Symptoms global score (r = .367, p =.007) and with a thought disorder subscale of the Brief PsychiatricRating scale (r = .334, p = .014). However, as expected, it wasnot correlated with the Schedule for Negative Symptoms globalscore (r = -.092, ns).

The Meyer data set allows us to address incremental validity.For the reasons given above, the only defensible hierarchicalanalytic strategy for this purpose is to add the EII and the otherRorschach variables after the MMPI variables. These analyses arepresented in Table 5. The correlation between EII and DxMax is.35. Given the limitations of DxMax as a measure of impairment,the incremental Rs for the EII and the Rorschach are quite impres-sive and may approach the maximum validity of DxMax as ameasure of impairment.

Conclusion regarding Dawes (1999). Thus, Dawes's (1999)analyses support the incremental and criterion validity of theRorschach. The EII provided significant unique predictive powerover MMPI variables for psychological impairment or disturbanceas grossly estimated by the maximum severity of diagnosis. Ana-lytic decisions by Dawes reduce the sensitivity of the analyses and,as a result, minimize the potential to support the Rorschach.Re-analyses of these data sets provide clear empirical support forcriterion validity and incremental validity of the EII and theRorschach.

Challenges to Incremental Validity of the Rorschach

Recent claims (Garb, 1999; Garb et al., 2001; Hunsley & Bailey,1999) that the Rorschach has poor incremental validity are basedlargely on old studies that confound clinical judgment with incre-mental validity (Garb, 1994; see Garb, 1998, for a reproduction ofthese studies), a point we return to later. These studies also do nothave the benefit of the CS scoring system and established inter-pretive routines, in addition to lessons learned from clinical judg-ment studies (Exner, 1993; Garb, 1994, 1998; Meyer et al., 1998).Accordingly, these old experiments now lack ecological validity.

Table 5Incremental Validity of Ego Impairment Index (EII) andRorschach From Dawes (1999) With Meyer and Resnick's(1996) Data Set Predicting Ratings of Severity of Impairment ofDx (DxMax)

Predictor

MMPIMMPI + EIIMMPI + Rorschach

R

.366

.424

.492

Change in R2

.134

.058

.108

Incremental rover MMPI only

.241

.329

Note. The Minnesota Multiphasic Personality Inventory (MMPI) and Ror-schach variables for analysis were those selected by Dawes: For the MMPI,mean elevation clinical scales and Goldberg Index; and for the Rorschach,EII, number of responses, and XQUAL.

Incremental Validity and Clinical Utility

The clinical utility of a test emerges within individual cases thatpose idiosyncratic questions and require idiographic decisions(Kleinmuntz, 1990; Meehl, 1957; Meyer et al., 1998). Anothercomplicating factor is that the "yield" from other sources of dataand tests varies from case to case. For example, a Rorschach maybe too sparse to adequately address interpersonal perception orthought disorder. In one case, the MMPI may be too exaggeratedto address depression. For these and other reasons, the usefulnessof a particular test varies greatly from case to case and question toquestion. "Field" incremental validity, therefore, only manifests inactual clinical practice in the form of clinical usefulness and theimpact of different tests in decision making within individualcases. Weiner (1999) and Strieker and Gold (1999) providedcompelling rationales and case examples that clearly demonstratethe clinical "field" incremental validity of the Rorschach in indi-vidual cases.

Future Research

As with all personality measures, more research that demon-strates incremental validity is needed. More sophisticated researchstudies might address complexes of Rorschach variables with reallife behavioral and cognitive criteria. It would also be helpful tolook at response styles, response sets, and contextual moderatorvariables that might influence incremental validity. In other words,Rorschach variables may generalize best to contexts that resemblethe Rorschach problem, such as complex situations in which thereis little external guidance so that the individual has to make his orher own way. Also, the adequacy of criterion variables and statis-tical power present technical challenges to such research.

Two Applications of Special Interest to Practitioners

Thought Disorder and Implications for the Course ofSchizophrenic Spectrum Disorder

A variety of Rorschach summary scores have traditionally beenused to assess and conduct research on thought disorder. Earlyresearch in this area used the Delta Index of Thought Disorderfrom which the Thought Disorder Index (Johnston & Holzman,1979) was developed. Both indexes share many features with CScognitive special scores and their weighted sum, the WSUM6.There are abundant data that support the irrefutable conclusion thatthese scales, as well as similar summary scores from the CS andother systems, are related to thinking disturbance, psychoses, andschizophrenic spectrum disorders (e.g., Acklin, 1999; Cadenhead,Perry, & Braff, 1996; Exner, 1991, 1993; Hilsenroth, Fowler, &Padawer, 1998; Kleiger, 1999; Perry & Braff, 1994; Perry, Geyer,& Braff, 1999; Perry, Moore, & Braff, 1995; Viglione, 1999;Weiner, 1966: Perry, Viglione, & Braff, 1992). In addition, re-search summarized below also associates Rorschach measureswith schizophrenic spectrum disorders, genetic and environmentalrisk factors, psychophysiological processes in schizophrenia, andsubsequent outcome.

Empirical findings. The Thought Disorder Index is associatedwith multiple criteria related to biological aspects of schizophreniaand psychosis proneness. Two studies associated the ThoughtDisorder Index with the Chapman Psychosis Proneness Scales


(Coleman, Levy, Lenzenweger, & Holzman, 1996; Edell & Chap-man, 1979), a measure of magical thinking, perceptual aberration,and social anhedonia (Chapman, Chapman, & Kwapil, 1995;Chapman, Chapman, & Raulin, 1976,1978; Eckblad, Chapman, &Chapman, 1982). In another study, similar Thought Disorder Indexelevations were produced by unaffected relatives of schizophrenics(Shenton, Solovay, et al., 1989). The Thought Disorder Index alsopredicted the development of psychotic and psychotic-like symp-toms over a 2 to 3 year period in a sample of Axis II and affectivedisorder adult patients over and above information from clinicalinterviews (O'Connell, Cooper, Perry, & Hoke 1989). InitialThought Disorder Index scores also demonstrated clinical utility topredict psychotic and psychotic-like symptoms at follow-up. TheThought Disorder Index was about three times greater amongpsychotic children and high-risk children compared with normalchildren (Arboleda & Holzman, 1985). In this study, ThoughtDisorder Index scores in nonpsychotic, hospitalized children werenot different from those of normal children. In a family adoptivestudy, Wahlberg et al. (1997) found that genetic and environmentalfactors interacted to produce elevated Thought Disorder Indexscores.

Empirical data also associate Rorschach measures of thoughtdisorder with more direct biological measures of brain functioning.For example, Daniels, Shenton, et al. (1988) found that patientswith right hemisphere damage displayed fragmented thinking,manic patients displayed playful thinking (as measured by sub-components of the Thought Disorder Index), and schizophrenicpatients used idiosyncratic thinking. A series of studies (Perry &Braff, in press; Perry & Braff, 1994; Perry et al., 1999) demon-strates relationships between (a) the EII and its subcomponents and(b) neurophysiological criterion measures of "gating." (Gating isthe ability to filter out irrelevant information in the environment,and it is considered to be a deficit in the neural circuitry ofschizophrenia.) In one study, these information-processing crite-rion variables and Rorschach responses were measured in a near-simultaneous computer paradigm (Perry et al., 1999). Information-processing deficits were not correlated with symptom rating scalesbut were correlated with form quality minus (X— %) and cognitivespecial scores (WSUM6). X—% independently accounted for 57%(p < .001) of the variance in the neurophysiological measure andcontributed 35% (p < .01) of the unique variance beyond theclinician-rated Schedule for Positive Symptoms Delusion subscalescore. The Schedule for Positive Symptoms Delusion scale wasselected among five symptom scales, because it had the greatestcorrelation with the neurophysiological measure. Thus, X—%demonstrated incremental validity over a variety of symptomscales. In this study the EII was not calculated because of therestriction of the number of responses, due to the demands of thecomputer-generated paradigm. Nevertheless, these results supportthe hypothesis that the EII and its subcomponents tap into thecognitive dysfunction that is observed among schizophreniapatients.

Challenges, conclusions, and future research. Garb et al.(2001) took exception to Viglione's (1999) claim that Rorschachvariables can be helpful in determining biological vulnerability forschizophrenia and that these variables have been linked to biolog-ical processes associated with schizophrenia. To do so, they iso-lated two individual studies from the mass of findings in theliterature. The research with various Rorschach measures of

thought disorder, with and outside of the CS, clearly reveals thatthe Rorschach can produce data related to the biological aspects ofthought disorder and psychosis proneness. Although the ThoughtDisorder Index administration typically differs from the CS ad-ministration and conclusive data need to be collected in regard tothe impact of these administrative differences, similar findingsacross the many Rorschach summary scores of thought disordersuggest that these positive findings should generalize to the EIIand other CS-based variables measuring disordered thinking. Ofcourse, research is needed to confirm this logical expectation.

In addition, further research should be conducted to specify theassociation between the Thought Disorder Index and CS summaryscores of thought disturbance, using protocols collected accordingto CS administrative procedures. Other research goals might be toconduct prospective, longitudinal studies to confirm that theseRorschach variables can predict (a) the emergence or return ofpsychoses, (b) negative outcomes in the course of schizophrenicspectrum disorders, and (c) the development of these disordersamong high-risk children. It would be useful to know whetherthese associations are moderated by Rorschach variables and col-lateral non-Rorschach information. For example, simple or preser-vative records may be associated with false-negative Rorschachthought disorder findings, given that the perseveration evident inRorschach records is poorly captured by existing scores. Improve-ments in the measurement of thought disorder within the CS arepossible (e.g., Kleiger, 1999; Perry, Potterat, Auslander, Kaplan, &Jeste, 1996), including the possibility of incorporating aspects ofthe Thought Disorder Index, adding the EII to the CS, developingmore meaningful measures of perseveration, and refining proce-dures for coding ALOG and DR so that they more selectivelyidentify problem-solving failures. This research should also ad-dress the sensitivity of Rorschach variables to problem-solvinglimitations in relatively well-functioning individuals.

Suicide

The CS S-CON is an actuarial variable that yields a single scoreto assess suicide risk from a composite of 12 variables and ratios.On the basis of all the available data and the cost-benefit offalse-positive versus false-negative errors, continued use of theS-CON is in order (Viglione, 1999). It should not be used to ruleout suicide risk, but it can be used to increase awareness aboutself-destructive behavior and suicide.

The Exner and Wylie (1977) study still qualifies as one of themore methodologically rigorous studies of suicide prediction. Inthe original study of 59 completed adult suicides, the authors founda total S-CON score of 8 positive indices correctly identified 75%of the patients who subsequently died as a result of their suicideattempt. Fowler, Piers, Hilsenroth, Hold wick, and Padawer (2001)examined the predictive relationship between the CS S-CON andlethality of suicide attempts within 60 days following administra-tion of the Rorschach. This study uses an ecologically valid andcompelling behavioral criterion. Patient records were reliably ratedas nonsuicidal (n = 37), parasuicidal (n = 37), or near lethal (n =30), based on the presence and lethality of self-destructive acts.The S-CON differentiated patients who would later engage innear-lethal suicide attempts from those who would engage inparasuicidal behavior and from those who did not engage inself-destructive behaviors. An S-CON score of 7 or more success-


fully predicted which patients would engage in near lethal suicidalactivity relative to parasuicidal patients (overall correct classifica-tion [OCC] rate = .79), nonsuicidal inpatients (OCC = .79), andcollege students (OCC = .89). Logistic regression analysis re-vealed that an S-CON score of 7 or more was the sole predictor(p < .01) of emergency medical transfers due to suicide attemptsfrom among nine demographic, IQ, psychiatric, and Rorschachpredictors. Some of these predictors included clinician-rated globalpsychiatric severity, GAP scores, and the presence of DSM-IVAxis I and II disorders. Thus, this study demonstrates the incre-mental validity of this Rorschach scale beyond clinician interviewand clinician ratings. The total S-CON score also predicted whichpatients would have a subsequent drug overdose (r = .32, p =.001) within 60 days of the Rorschach administration.

To challenge the S-CON, Garb et al. (2001) focused on twostudies that used postattempt suicide criteria (Meyer, 1993; Silberg& Armstrong, 1992) and one presentation that has been describedonly in reviews by Eyman and Eyman (1987, 1991, 1992). How-ever, direct examination of these reviews reveals empirical supportfor the efficacy for two single sign suicide predictor variables—transparency responses and color-shading blends. These scores areincorporated into shading, form dimension, vista, and color-shading subcomponents of the S-CON.5 The Rorschachs in theEyman and Eyman study were administered in the Rapaport-Schafer system (Rapaport, Gill, & Schafer, 1946; Schafer, 1954)but recoded in the CS. The Rapaport-Schafer system does notcontain all the same variables as the CS so that inquiry questionsdo not elicit all relevant information. Translation of these protocolsto CS variables on an actuarial basis with specific cutoffs may notbe justified without further research.

As a whole, the empirical data support the continued use of theS-CON. The decision-making utility of the S-CON may not havebeen exhaustively evaluated, but based on substantial positiveresults one cannot ignore the S-CON data if the false-negativeerror means losing a life. The most enlightened clinical interpre-tation of an elevated S-CON is to assert that the Rorschachindicates additional suicide risk that might not be recognizedthrough other means. Practice ethics and an S-CON of 7, 8, ormore should lead clinicians to consider suicidal risk and to collectadditional information about self-destructive risk. Of course, theS-CON may produce false positives in some contexts because ofthe low base rates of actual suicide (Finn & Kamphuis, 1995). Inreal-life clinical decision making, which takes into consideration awide range of contextual and collateral information, the cost of thisfalse-positive error is minimal compared with the cost of false-negative error (i.e., missing an imminent suicide).

Clinician Judgment

Ecologically Valid Clinical-Actuarial Judgment

The goal of a large portion of the decisions in formulating theadministration procedures of the CS was to minimize examinerand contextual effects identified in earlier research (Exner, 1974).Interrater reliability and temporal consistency data summarizedearlier in this article, as well as other research data (e.g., Brinder-son, 1995; Cheyette, 1992; Perry, Sprock, Schaible, McDougall, etal., 1995; Viglione & Exner, 1983), suggest that these problemshave been minimized. Similarly, a large portion of the interpretive

tactics and routines, particularly the interpretive search routines,were developed to minimize problems identified in clinical judg-ment studies.

The current CS combination of actuarial and clinical methods(Exner, 1991, 1993; Strieker & Gold, 1999; Weiner, 1999) isconsistent with thinking in the field of judgment and decisionmaking (e.g., Garb, 1989). This clinical-actuarial method usesprocedures whose goal is to minimize cognitive judgment errorsand distortions.6 These procedures are probably ideal in addressingunique questions in clinical contexts where no two cases are alike.In contrast to the dated clinician judgment analogue research, inwhich the participant clinician interprets a limited, predefined, andidiosyncratic set of data, the clinician in the field uses data todevelop, to reformulate, and to refine referral questions. Theclinician also uses expertise to gather or to select accurate datarelevant to those questions. In doing so, the examiner constantlyinteracts with emerging data sets, to change existing questions,probabilities, and base rates (Meyer et al., 1998). Along the way,the examiner brings extreme base rates nearer to levels in whichtesting data would be more efficient in decision making. In thatsense, the single event, low base analogy found in the literature, forexample, "Is this client thought disordered?" as addressed byHunsley and Bailey (1999, p. 272), is an oversimplified analogueto clinical decision making in the field.

In actual clinical practice, decisions with low base rate phenom-ena rest on the consideration and synthesis of many small datapoints. An example of such a situation may be the evaluation of aschizophrenic-like thought disorder in outpatient or forensic con-texts. Synthesizing such data as history, symptoms, observations ofthought disorder, and related characteristics can eventually lead tooptimal base rates closer to 50% (Hunsley & Bailey, 1999). Inother words, in a patient without the history and observations thatmight lead to a sufficiently high probability of diagnosing aschizophrenic-like thought disorder, the clinician with positiveevidence of confused thinking on the Rorschach might search forhistorical or behavioral information consistent with (a) excessivelyelaborated, idiosyncratic, and overly abstract thinking (Franklin &Cornell, 1997; Gallucci, 1989) or (b) thinking problems associatedwith intrusions of traumatic imagery (Holaday & Whittenberg,1994; Sloan, Arsenault, Hilsenroth, Harvill, & Handler, 1995).These are alternative conditions that may lead to indices of con-fused thinking on the Rorschach that have been supported byresearch.

Such a clinical-actuarial model is consistent with the thresholdmodel described by Rogers (1997), which mitigates errors inevaluating malingering. It also explains why configurational, syn-thetic approaches, as described by Strieker and Gold (1999) and asdemonstrated by Weiner (1999), are fundamental to the clinical-actuarial method. In this form, this method minimizes the possi-bility of a false-positive actuarial interpretation. This clinical-

5 The Eyman and Eyman study was not included in the journals identi-fied by Meyer (1999) and was not published as a research report. Thus,Viglione (1999) did not consider it according to the predetermined, objec-tive inclusion criteria established for his review.

6 There is no a priori reason that this actuarial-clinical—nomothetic-idiographic approach would have to be confined to CS variables. Othervariables with a sound research basis can certainly be added.


actuarial method with the Rorschach uses an ecologically valid andinformed understanding of interactive probabilities to increaseaccuracy of in vivo decision making in applied psychology.

Meehl and Rosen (1955) addressed this issue in the form of the"successive-hurdles" approach. Their example considered separatetests for a brain tumor decision, and they described an increase inhit rates. Meehl and Rosen reported small relative increases in hitrates over base rates and large relative drops in false-negativeerrors with this procedure. For example, successive tests with (a)90% true positive and 15% false positive for Test 1, followed by(b) Test 2 with 80% true positive and 10% false positive wouldincrease hit rates from 90% to 96%. In this example, overall errorsare cut by more than half, and false-negative errors are reduced bymore than three times, from 10% to 3%. The improvement inclinical utility is a function of the cost of these false-positive andfalse-negative errors. In psychological evaluation, this processcontains many more interactive steps. Thus, illness course, symp-tom report, Rorschach scores for thought disorder, observations ofthought-disordered behavior in interview, and other Rorschachvariables and observations come together. On the basis of each ofthese "tests," the examiner chooses to observe more data to con-tinue to test the hypotheses. Both a yes-no decision and anidiographic, qualitative description of the problem emerge in thetypical clinical or forensic Rorschach application.

The clinical-actuarial judgment method is further complicatedwhen one adds the strengths, weaknesses, and predilections of theindividual examiner (Strieker & Gold, 1999). Along these lines,and in response to criticisms by Wood, Lilienfeld, Garb, andNezworski (2000), Garfield (2000) emphasized the skill of theclinician as an important determinant when using the Rorschach inapplied practice. To study these issues, more creative and ecolog-ically valid ways of studying the clinician and improving in vivoapplied judgment need to be developed both within the quantita-tive tradition, but also in terms of qualitative research (e.g.,Schaich, 1999). This approach is consistent with the APA Psycho-logical Assessment Work Group report (Meyer et al., 1998.)7

Clinical Judgment: Future Research

Further research on clinical judgment is needed not only for theRorschach but also for other types of decision making. It might beilluminating to conduct new studies with the CS and non-CSvariables, using clinicians who are aware of errors and pitfalls inclinical judgment. As Garb (1998) stated, "most of the judgmentstudies on the use of projective tests are over 20 years old" (p.232), with virtually no studies with the modern CS and otherRorschach variables. Such analogue clinical judgment researchwould have to meet ecological validity challenges, namely, how toreproduce the clinical-actuarial method in the lab, how to motivateclinicians and enforce the real life cost-benefit ratios for accurateversus inaccurate judgments, and how to make the questions anddecisions sufficiently idiographic, meaningful, and malleable. De-signs must also incorporate better criterion measures. Reproducingthe cogwheeling interactive approach (Meyer et al., 1998), vicar-ious functioning of the respondent and interpreter in facing un-bounded problems (Hammond, 1955; Levy, 1963) in an ecologi-cally valid way is critically important.

Lack of ecological validity, poor criteria, and the failure tomotivate clinicians all reduce potential effect sizes. If incremental

validity is to be addressed quantitatively, an additional challenge ispower. Sample sizes must be very large or multiple clinical deci-sions must be evaluated to be able to rule out false-negative results.

Utility, Cost-Benefit, and the Rorschach

The Rorschach allows us to study samples of behavior collectedunder similar conditions in different native languages around theworld (Erdberg & Schaffer, 1999). There is an efficiency tosampling behaviors with a single technique to develop one'sunderstanding and appreciation of developmental changes acrossthe age span and across all types of disorders and problems.

To make cost-benefit comparisons ecologically valid, one wouldhave to envision a full range of costs and benefits equivalent to theRorschach. The alternative to the Rorschach would have to addressa wide range of clinical, personality, forensic, developmental,cognitive, and neuropsychological constructs and applicationsacross the life span and around the world. The Rorschach allowsone to develop a common, experientially based database for theproblem-solving practices and predilections of clients from age 5through old age.8 One could identify many other single purposescales within a specific content-construct area that might competewell with the Rorschach, but to produce the same benefits, onewould have to master and monitor developments in perhaps 50alternative instruments. Another cost is the expense and hassle ofpurchasing kits, test blanks, and computer programs and payingper use fees. Also, the Rorschach provides an efficient way tocollect a behavior sample outside of interview behavior. Psychol-

7 In addressing clinical utility, Garb et al. (2001) presented a factuallyincorrect and misleading picture of Viglione's (1999) review of the liter-ature on clinician judgment: "Half of the studies cited by Viglione werepublished in the 1960s and 1970s, with the remainder of the studiespublished in the 1980s" (p. 437). Viglione cited five research reports,which met a priori inclusion and exclusion criteria, providing evidence thatclinicians can make valid and useful judgments from Rorschach data(Bilett, Jones, & Whitaker, 1982; Cerney, 1984; Dana & Back, 1983;Dudek & Marchand, 1983; Holzman et al., 1974). Viglione cited threeadditional works, not as research reports, but merely as descriptions forscales used in the five cited studies: (a) Urist (1977) for the Mutuality ofAutonomy Scale, (b) Johnston and Holzman (1979) for the ThoughtDisorder Index, and (c) Masling, Rabid, and Blondheim (1967) for theRorschach Oral Dependency Scale. Garb et al. (2001) argued that Viglione(1999) consistently ignored negative findings and then listed seven studiessupposedly ignored by Viglione (Albert, Fox, & Kahn, 1980; Bilett et al.,1982; Cochrane, 1972; Gadol, 1969; Golden, 1964; Perez, 1976; Turner,1966). Only the first two of the studies were among the 445 studies selectedfor review by the editor of this Special Series. These studies addressed theincremental validity of judgments and the relationship between the validityof judgments and confidence or training, rather than simply the issue ofjudgment. Garb et al. (2001) also asserted that "in fact, positive results havenever been obtained for the Rorschach in studies on clinical judgment and[italics added] incremental validity, confidence and [italics added] validity,and training and [italics added] validity" (p. 437). To clarify this statement,no single Rorschach study simultaneously offers (a) a demonstration ofvalid clinician judgment with incremental validity techniques, (b) a dem-onstration of an association between judges' avowed confidence and ac-curacy of judgments, and (c) a demonstration of an association betweenjudges' training and accuracy of judgment.

8 The Rorschach has applications below age 5 for testing developmentalissues (Leichtman, 1996).


ogists' time is valuable, but so is the client's time; therefore, it isanother cost to be entered into cost-benefit analysis. Mastering onetest, the Rorschach, is efficient because it eliminates the need formany content specific tests. As a result, we can pick and choosemore judiciously and master select instruments within an assess-ment battery, rather than misusing a large number of tests for allthe potential purposes that the Rorschach addresses.

Use of the Rorschach: Past, Present, and Future

Training, Research, and Clinical Use

Repeated surveys of psychological test use over the past 40years have shown a substantial, consistent, and sustained use of theRorschach in academic training, research, and clinical settings.Surveys consistently indicate that over 80% of graduate programsteach the Rorschach and that students regard this training asimportant in developing other clinical skills and in understandingtheir patients better, and as useful in their practicum-intemshiptraining (e.g., since 1995, Camara, Nathan, & Puente, 2000;Hilsenroth & Handler, 1995; Watkins, Campbell, Nieberding, &Hallmark, 1995). Although some surveys report that the Rorschachtakes more of the clinician's time than many other tests, they alsoreport that it is one of the most frequently used (e.g., Camara et al.,2000).9

Also, 90% of clinical practitioners working in the field ex-pressed a belief that clinical students should be competent inRorschach assessment (Watkins et al., 1995). This attitude wasshared by internship directors (Durand, Blanchard, & Mindell,1988). In a recent but relatively small survey of 84 (19%) of the445 APA-approved internship programs in 1997 (Piotrowski &Belter, 1999), internship directors reported that the MMPI-2/MMPI-A, WAIS-R/WAIS-III, and Rorschach were the top threeassessment instruments used in internship programs and wereconsidered essential for practicing psychologists. Twenty-seventraining directors reported decreased emphasis on projective as-sessment, whereas 12 reported an increased emphasis. Also, hoursof seminar time required by internship programs were virtuallyidentical for projective and self-report assessment measures. Fi-nally, the Rorschach was used more than all five of the behavioralassessment instruments listed by Piotrowski and Belter (1999). Arecent survey of practitioners' test use with adolescents suggeststhat testing has declined because of managed care constraints, butthat the Rorschach has retained its status as the second most usedinstrument (Archer & Newsom, 2000).

The two most comprehensive recent surveys of predoctoralinternships, one including 329 (Stedman, Hatch, & Schoenfeld,2000) and another including 324 (Clemence & Handler, 2001)Association of Psychology Postdoctoral and Internship Centerssites (most of which were programs accredited by the APA),revealed that internship training directors greatly value the Ror-schach as well as integrated test batteries. Again, training directorsreported a desire for incoming interns to have had courses or agood working knowledge of the Rorschach. Academic depart-ments that are making decisions about the place of the Rorschachin their curriculum should closely attend to these data.

The Rorschach is the second most frequently researched per-sonality assessment instrument. This consistently high rate ofempirical investigation led the authors of a recent review to state

the following: "Whether viewed from the perspective of researchattention or practical usage, the Rorschach inkblot technique con-tinues to be among the most popular personality assessment meth-ods and predictions about the technique's demise would appearboth unwarranted and unrealistic" (Butcher & Rouse, 1996, p. 91).

Forensic Use

Weiner, Exner, and Schiara (1996) reviewed responses from 93practioners who testified in over 4,000 criminal cases, over 3,000custody cases, and 858 personal injury cases, for a total of almost8,000 cases. In only 6 of these cases (0.08%) did these respondentsreport that the integrity of the test was challenged and in only 1case (0.01%) was it ruled inadmissible. Although Weiner's sam-pling technique may have limitations, these data provide a "goodempirical basis for concluding that, contrary to whatever opinionmay be voiced, the Rorschach is welcome in the courtroom" (pp.423-424).

To address the question of the legal weight, or the effect theRorschach has on court judgments, Meloy, Hansen, and Weiner(1997) reviewed court of appeals citations from 1945 to 1995. In90% of the 194 cases, "the admissibility and weight of Rorschachdata were not questioned by either the appellant or the respondentand were important enough to be mentioned and discussed in thelegal ruling by the court of appeal" (p. 60). Typically whenquestions arose, the practitioner's interpretation or findings de-rived from the test, rather than the test itself, were challenged. Inonly two cases (0.8%) was the test criticized as a psychologicalmeasure. These data demonstrate that the Rorschach "has author-ity, or weight, in higher courts of appeal in the United States" (p.61). hi a systematic, comprehensive legal analysis of the admissi-bility of the Rorschach, McCann (1998) agreed with the courts. Heconcluded that a focus on the structural summary from the CS"meets professional and legal standards for admissibility of psy-chometric evidence and expert testimony" (p. 140).

Challenges

In an earlier article in this Special Series, Dawes (1999) referredto the "deficiency" (p. 301) of the Rorschach in relation to thecriterion of reasonable certainty used in forensic applications. Hisargument, concerning the forensic use of the Rorschach, appears tobe based on some elusive juxtaposition of incremental validitystatistical analysis with the forensic notion that expert testimonyshould be "incremental" in the forensic sense of aiding the trier offacts. Contrary to his argument and his own data, the evidencefrom the courts suggests that more direct self-report assessmentmethods may not be as valuable in forensic adversarial evaluationcontexts; therefore, the incremental value of the Rorschach interms of assisting the trier of facts increases. In fact, Dawes's ownfindings indicated that whatever deficiency the Rorschach pos-sessed, the MMPI also possessed because the Rorschach improvedon predictions made by the MMPI.

Given these data, the recent task force report by Division 12(Clinical) of the APA (1999), concerning the development of a

9 Citations before 1995 that support the assertions in this section areexcluded for the sake of brevity.


model assessment curriculum for graduate training in assessment,is confusing and misguided. In this report, Rorschach training wasranked 95th in importance out of the total 105 topics. The availablereliability and validity data, coupled with other contemporarysurvey material reported earlier, suggest that there is vast discrep-ancy between the Division 12 Task Force future model curriculumfor assessment and actual graduate training needs. Should thesechanges in graduate assessment training occur, students and futureclients will have to pay for the misconceptions of graduate faculty.

Also Hunsley and Bailey (1999) cited a dated survey (Wade &Baker, 1977) of practicing clinicians to claim that practitionersdisregard scientific evidence. Actually, there is empirical evidenceto show quite the contrary. Beutler, Williams, Wakefield, andEntwistle (1995) recently found that clinicians in private practicevalue research and put forth effort to understand scientific find-ings. There is no empirical evidence that indicates that the mannerin which psychologists typically use the Rorschach is differentfrom the standardized methods in which they were trained, and noevidence that psychologists who are using the Rorschach areunaware of its research findings.

Conclusion

The data on the use and perceived value of the Rorschachclearly demonstrate that psychologists find the Rorschach to be ofvalue in routine clinical practice. Some psychologists ask why somany practitioners persist in their use of the Rorschach despitecurrent time pressures and limited fees by third party payers. Thisbecomes even more perplexing when self-report questionnairesand structured interviews may require less of the clinician's time.Considering the persistent use of the Rorschach in the context ofthe available reliability and validity information leads to someobvious conclusions: One must speculate that clinicians and thosein charge of clinical training find it helpful because of its reliabilityand validity relevant to challenges in applied settings. In otherwords, clinicians continue to use the test despite its time require-ments because of its clinical utility.

A substantial body of research demonstrates that individualsexhibit a defensive bias and manipulate their responses whenasked to self-report aspects of their personality or psychopathol-ogy. For example, individual differences in self-serving biaseshave been demonstrated in relation to attributions, self-descriptions, inferences, personality traits, and avoidance of neg-ative affect. Rorschach research supports the view that the Ror-schach is most useful in (a) contexts in which the respondent maybe unwilling or unable to self-report problems and (b) those inwhich clinicians are trying to predict real-life behavior (Hiller etal., 1999; Viglione, 1999). In addition to routine clinical contexts,the data suggest that the Rorschach may be especially useful underconditions that might induce response manipulation in self-report.These might include employment, forensic, and custody evalua-tions, and other settings that involve adversarial examiner-respondent relationship components (Rogers, 1997).

Methodological Challenges to Validity of the Rorschach

Challenges

Rorschach researchers (Exner, 1995; Viglione, 1997) have of-fered recommendations to improve Rorschach research and, in

turn, improve the validity and utility of the test. Others have alsorepeatedly offered methodological recommendations and stan-dards for Rorschach research, and they have used these standardsto reject findings from individual studies supportive of the Ror-schach (Garb, 1999; Garb et al., 2001; Hunsley & Bailey, 1999;Wood et al., 2000; Wood, Nezworski, & Stejskal, 1996a, 1996b).These same standards are not fully applied when evaluating studiescited as yielding negative evidence for the Rorschach. Also, im-partiality would require applying the same standards to all tests,rather than reserving them specifically for the Rorschach (Garfield,2000). In the first set of publications for this Special Series inPsychological Assessment, Viglione (1999) met a number of thechallenges posed by Rorschach critics. Nevertheless, the positiveresearch findings summarized by Viglione (1999) were met withthe same repeated methodological criticisms (Garb et al., 2001).Similarly, other research reports (e.g., Meyer, 1997a; Hilsenroth,Fowler, Padawer, & Handler, 1997) have not been recognized aspositive responses to criticisms. In the section below, we examineGarb, Wood, and their colleagues' applications of these criticismsabout methodology in Rorschach research.

Within and Across Diagnoses

Viglione's (1999) synthesis of the empirical research supportsthe conclusion that "the Rorschach is useful in examining manyclinically relevant criteria and behavior within and across diag-noses" (p. 260). However, in reaction to Strieker and Gold's(1999) summary of the literature, endorsing the usefulness of theRorschach in evaluating depressed, schizophrenic, or acutely sui-cidal patients, Garb et al. (2001) asserted that "empirical researchsuggests that the Rorschach is not useful for all these tasks" (p.433) and cited only Wood et al. (1996a, 1996b) to bolster thisstatement. This claim overlooks a great deal of evidence in supportof the usefulness of the Rorschach in these disorders and associ-ated phenomena. There are numerous examples of research thatsupport the validity of the Rorschach applied to various problemswithin and across diagnoses (e.g., see Viglione, 1999).

Empirical Data in Peer-Reviewed Journals, Accessible toAll

Some critics have maintained that too much of the empiricalsupport for the Rorschach contained within studies in Exner'svolumes is not published in peer-reviewed journals and is thereforenot available for review, In response to this criticism, Viglione(1999) synthesized research findings from peer-reviewed journalsonly. He reported considerable empirical support for the test.Viglione focused on the Methods and Results sections, rather thanon the theorizing of the authors. Thus, a careful review of theempirical data published in peer-reviewed journals supports theutility of the Rorschach.

Selection of Studies

Rorschach critics (e.g., Garb et al., 2001; Wood et al., 2000;Wood, Nezworski, Stejskal, Garven, & West, 1999) have claimedthat Rorschach researchers selectively present findings. Meyer(1999) requested that the contributors of this Special Series ad-dress 445 articles in five targeted journals published between 1977


and 1997. In response, Viglione (1999) applied objective, meta-analytic type selection criteria to select 128 research reports. Incontrast, Hunsley and Bailey (1999) confined their review to 28studies, whereas Garb et al. (2001) reviewed only 44 of these same445 articles. In addition, Viglione's selection criteria resulted inthe exclusion of supportive Rorschach evidence found not only inExner's texts, but also in review articles, meta-analyses, and books(e.g., Blatt et al., 1994; Exner, 1993; Gacono & Meloy, 1994;Kleiger, 1999; Masling, 1986; Weiner, 1966). Most of the negativestudies selected by Garb, which Viglione was said to have ignored,were not among those identified by the series editor. On the otherhand, Garb et al. uncovered virtually no research supportive of theRorschach, even though they were provided with the same 445target articles as the other series contributors. Viglione includednegative findings when they occurred.10

Experiment-Wise Alpha Levels

Garb et al. (2001) maintained that the Rorschach literature failsto control experiment-wise alpha levels. For example, Garb et al.(2001) viewed Lemer and St. Peter's (1984a, 1984b) positivefindings about borderline personality structure and Rorschach hu-man representations as a flawed study because of the failure tocontrol for false-positive findings. Garb et al. (2001) did not reportthat 41 of 64 analyses were significant at p < .05, and 33 of 64were significant at p < .01.n Mathematically, these positiveresults cannot be explained by chance. All other analyses werelegitimate, post hoc group contrasts for the 41 significantcomparisons.

Criterion Contamination and Diagnosis

Critics have faulted Rorschach researchers for using cliniciandiagnosis as a criterion (Garb et al., 2001; Wood et al., 2000). Forexample, Garb (1998) questioned the use of clinicians as criterionjudges: "Historically, the clinicians making the indicator ratingshave been called criterion judges and their ratings criterion ratings,but these terms are inaccurate because the ratings are fallible" (p.14). Nonetheless, these same critics have emphasized studies byArcher and his colleagues (Archer & Gordon, 1988; Archer &Krishnamurthy, 1997) to argue against the incremental validity ofthe Rorschach. The Archer and Krishnamurthy (1997) study usedthe clinical diagnosis from just one clinician as the sole basis forthe diagnosis. Moreover, Archer and Gordon (1988) noted that"psychological test results were frequently available and poten-tially involved in the diagnostic decisions" (p. 279) so that crite-rion contamination and bias are possible. Both studies lack reli-ability statistics for their diagnostic criteria. These studies do notmeet the methodological standards imposed by the critics onstudies with evidence supportive of the Rorschach. Yet, thesestudies are cited as negative evidence for the validity of theRorschach. This is clearly evidence of a double standard used bythe critics of Rorschach research.

Implications and Conclusions

We have addressed the Rorschach's strengths and weaknessesand offered research recommendations in 10 focal areas relevant tothe clinical utility of the test. In addressing these issues, it is our

intention to give readers an informed appreciation for the substan-tial but often overlooked research basis for the reliability, validity,and utility of the Rorschach. From the broadest point of view, wehave demonstrated that examining the empirical findings can onlylead to the conclusion that the Rorschach is a valuable test (Vigli-one, 1999; Weiner, 1996). Interrater reliability by well-trainedcoders both in the field and in the lab are very strong. Temporalconsistency data are good, as are the currently available normativedata, though both could be improved. Criterion validity for the testis very strong, and incremental validity and meta-analytic data arepresent with more data emerging. The current sweeping criticismsof the Rorschach are largely without merit, uninformed, and biasedagainst the test. Thus, the Rorschach, by virtue of the accumulatedresearch, is now "better prepared to stand the tests of reliability andvalidation" (Exner, 1974, p. 16).

Clinicians persist in using the test because it continues to pro-vide comprehensive and valuable information about their patients.Thus, these clinicians are living proof of the application of theseresearch findings. In turn, the research findings mandate that thetest continue to be used. We have broadened the scope of thecost-benefit issues and utility analysis in the application of the testto be more consistent with clinical practice. In doing so, we haveidentified other obvious benefits to the Rorschach. These includethe use of a single test for many applications and the opportunityto collect quantifiable data while simultaneously collecting a be-havioral sample in a context that maximizes individual differencesin problem solving. The Rorschach serves those who are interestedin having a valuable tool to help understand an individual com-prehensively and to synthesize information across many domains(Strieker & Gold, 1999). Undoubtedly, the behavior samplingopportunity that occurs during administration promotes these goalsand provides another foundation to generalize outside of the con-sulting room.

Acclaim for the Rorschach has elicited criticism (Wood &Lilienfeld, 1999). Recent critical commentaries of the Rorschachhave identified some areas (e.g., normative data and methodolog-ical problems in some studies) in which the Rorschach can beimproved. Like other tests, the Rorschach has its weaknesses.However, these recent criticisms are undermined by methodolog-ical double standards, confirmatory bias, incomplete coverage ofthe literature, failure to integrate positive contributions over thelast 5 years that have specifically addressed earlier criticisms fromthese authors, and a lack of any original data to support theirpositions. For example, calls for a moratorium on the use of theRorschach (Garb, 1999; Garb et al., 2001) clearly contradict em-pirical findings. Some clinical utility criteria (Hunsley & Bailey,1999) set such a high and unrealistic standard that implementingthem would result in abandoning all testing, interviews, and ob-servation for assessment. Useful criticism and reasonable clinicalvalidity criteria challenge future research rather than mandate our

10 Viglione (1999) cited 19 studies on pages 254-259 with negativefindings.

11 Viglione (1999) is criticized for not including Lerner and St. Peter's(1984a) first article. To be factual, the earlier article was published inPsychiatry, a journal excluded from the series editor's targeted list. It wasnever considered for inclusion, according to the objective inclusion criteriaadopted in this review.


forsaking psychological and neuropsychological assessment alto-gether. However, an important contribution from critical reviewshas been to improve Rorschach research (Acklin, 1999), some-thing that is happening much more quickly than was the case afterRorschach researchers called for similar improvements a few yearsago (Exner, 1995; Viglione, 1997).

The informed use of the Rorschach will lead to a more sensitiveand helpful understanding of our clients as well as more effectiveinterventions. Continuing research and development of the test aswell as the provision for sound training and continuing educationin the Rorschach method, as with any assessment measure, are alsoessential. Nevertheless, the extant empirical research findings ne-cessitate that any scholar must accept the fact that the Rorschachtest method, its administration procedures, and its objective rulesfor quantifying behavior produce valid data and observations.

References

Acklin, M. W. (1999). Behavioral science foundations of the Rorschachtest: Research and clinical applications. Assessment, 6, 319-324.

Acklin, M. W., McDowell, C. J., & VerscheU, M. S. (2000). Interobserveragreement, intraobserver reliability, and the Rorschach ComprehensiveSystem. Journal of Personality Assessment, 74, 15—47.

Adair, H. E., & Wagner, E. E. (1992). Stability of unusual verbalizationson the Rorschach for outpatients with schizophrenia. Journal of ClinicalPsychology, 48, 250-256.

Albert, S., Fox, H. M., & Kahn, M. W. (1980). Faking psychosis on theRorschach: Can expert judges detect malingering? Journal of Person-ality Assessment, 44, 115-119.

American Psychiatric Association. (1994). Diagnostic and statistal manualof mental disorders (4th ed.). Washington, DC: Author.

American Psychological Association, Division 12, Presidential Task Force(1998). Assessment for the twenty-first century: A model curriculum.The Clinical Psychologist, 52, 10-15.

Anastasi, A., & Urbina, S. (1996). Psychological testing (7th ed.). NewYork: Macmillan.

Arboleda, C., & Holzman, P. S. (1985). Thought disorder in children at riskfor psychosis. Archives of General Psychiatry, 42, 1004-1013.

Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices ofschizophrenic and depressive diagnoses among adolescent inpatients.Journal of Personality Assessment, 52, 276-287.

Archer, R. P., & Krishnamurthy, R. (1997). MMPI-A and Rorschachindices related to depression and conduct disorder An evaluation of theincremental validity hypotheses. Journal of Personality Assessment, 69,517-533.

Archer, R. P., & Newsom, C. R. (2000). Psychological test usage withadolescent clients: Survey update. Assessment, 7, 227-235.

Atkinson, L., Quarrington, B., Alp, I. E., & Cyr, J. J. (1986). Rorschachvalidity: An empirical approach to the literature. Journal of ClinicalPsychology, 42, 360-362.

Beutler, L. E., Williams, R. E., Wakefield, P. J., & Entwistle, S. R. (1995).Bridging scientist and practitioner perspectives in clinical psychology.American Psychologist, 50, 984-994.

Bilett, J. L., Jones, N. F., & Whitaker, L. C. (1982). Exploring schizo-phrenic thinking in older adolescents with the WAIS, Rorschach andWISC. Journal of Clinical Psychology, 38, 232-243.

Blais, M., Hilsenroth, M., Castlebury, F., Fowler, C., & Baity, M. (2001).Predicting DSM-W Cluster B personality disorder criteria from MMPI-2and Rorschach data: A test of incremental validity. Journal of Person-ality Assessment, 76, 150-168.

Blatt, S. J., Ford, R. Q., Berman, W. H., Jr., Cook, B., Cramer, P., &Robins, C. E. (1994). Therapeutic change: An object relations perspec-tive. New York: Plenum.

Bornstein, R. F. (1996). Construct validity of the Rorschach Oral Depen-dency scale: 1967-1995. Psychological Assessment, 8, 200-205.

Bornstein, R. F. (1999). Criterion validity of objective and projectivedependency tests: A meta-analytic assessment of behavioral prediction.Psychological Assessment, 11, 48-57.

Bornstein, R. F., Bowers, K. S., & Robinson, K. J. (1997). Differentialrelationships of objective and projective dependency scores to self-reports of interpersonal life events in college student subjects. Journal ofPersonality Assessment, 65, 255-269.

Brinderson, C. (1995). Apparent nonpatient psychopathology on the Ror-schach: Disturbance or disinhibition? Unpublished doctoral disserta-tion, California School of Professional Psychology, San Diego, CA.

Burns, B., & Viglione, D. J., Jr. (1996). The Rorschach Human ExperienceVariable, interpersonal relatedness, and object representation in nonpa-tients. Psychological Assessment, 8, 92-99.

Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A. M., &Kaemmer, B. (1989). MMPI-2: Manual for administration and scoring.Minneapolis: University of Minnesota Press.

Butcher, J. N., & Rouse, S. (1996). Personality: Individual differences andclinical assessment. Annual Review of Psychology, 47, 87-111.

Cadenhead, K. S., Perry, W., & Braff, D. L. (1996). The relationship ofinformation-processing deficits and clinical symptoms in schizotypalpersonality disorder. Biological Psychiatry, 40, 853-858.

Camara, W., Nathan, J., & Puente, A. (2000). Psychological test usage:Implications in professional use. Professional Psychology: Researchand Practice, 31, 141-154.

Ceraey, M. S. (1984). One last response to the Rorschach test: A secondchance to reveal oneself. Journal of Personality Assessment, 48, 338-344.

Chapman, L. J., Chapman, J. P., & Kwapil, T. R. (1995). Scales for themeasurement of schizotype. In A. Raine. T. Lencz, & S. A. Mednick(Eds.), Schizotypal personality (pp. 79-106). New York: CambridgeUniversity Press.

Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1976). Scales forphysical and social anhedonia. Journal of Abnormal Psychology, 85,374-382.

Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1978). Body-imageaberration in schizophrenia. Journal of Abnormal Psychology, 87, 399-407.

Cheyette, L. (1992). The effects of examiner style on Rorschach test resultsof children with borderline psychopathology. Unpublished doctoral dis-sertation, California School of Professional Psychology, San Diego, CA.

Clemence, A., & Handler, L. (2001). Psychological assessment on intern-ship: A survey of training directors and their expectations for students.Journal of Personality Assessment, 76, 18-47.

Cochrane, C. T. (1972). Effects of diagnostic information on empathicunderstanding by the therapist in a psychotherapy analogue. Journal ofConsulting and Clinical Psychology, 38, 359-365.

Coleman, M. J., Levy, D. L., Lenzenweger, M. F., & Holzman, P. S.(1996). Thought disorder, perceptual aberrations, and schizotypy. Jour-nal of Abnormal Psychology, 105, 469-473.

Colligan, R. C., Osbome, D. M. S. W., & Offord, K. P. (1984). The agingMMPI: Development of contemporary norms. Mayo Clinic Proceed-ings, 59, 377-390.

Cooper, S. H., Perry, J. C., & O'Connell, M. (1991). The RorschachDefense Scales: II. Longitudinal perspectives. Journal of PersonalityAssessment, 56, 191-201.

Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. (1975). A MMPIhandbook: Vol. 11: Research application. Minneapolis: University ofMinnesota Press.

Dana, R. H., & Back, B. R. (1983). The concurrent validity of childRorschach interpretations. Journal of Personality Assessment, 47, 3-6.

Daniels, E. K., Shenton, M. E., et al. (1988). Patterns of thought disorder


associated with right cortical damage, schizophrenia, and mania. Amer-ican Journal of Psychiatry, 145, 944-949.

Dawes, R. M. (1999). Two methods for studying incremental validity of aRorschach variable. Psychological Assessment, 11, 297-302.

Dudek, S. Z., & Marchand, P. (1983). Artistic style and personality increative painters. Journal of Personality Assessment, 47, 139—142.

Durand, V., Blanchard, E., & Mindell, J. (1988). Training in projectivetesting: Survey of clinical training directors and internship directors.Professional Psychology: Research and Practice, 19, 236-238.

Eckblad, M. L., Chapman, L. J., & Chapman, J. P. (1982). The RevisedSocial Anhedonia Scale. Unpublished manuscript.

Edell, W. S., & Chapman, L. J. (1979). Anhedonia, perceptual aberration,and the Rorschach. Journal of Consulting and Clinical Psychology, 47,377-384.

Erdberg, P., & Schaffer, T. W. (1999, July). International symposium onRorschach nonpatient data: Findings from around the world. Paperpresented at the International Congress of Rorschach and ProjectiveMethods, Amsterdam, the Netherlands.

Erstad, D. (1996). An investigation of older adults' less frequent humanmovement and color responses on the Rorschach. Unpublished doctoraldissertation, Marquette University.

Exner, J. E. (1974). The Rorschach: A Comprehensive System (Vol. 1).New York: Wiley.

Exner, J. E. (1991). The Rorschach: A Comprehensive System: Vol. 2.Interpretation (2nd ed.). New York: Wiley.

Exner, J. E. (1993). The Rorschach: A Comprehensive System: Vol. 1.Basic foundations (3rd ed.). New York: Wiley.

Exner, J. E. (Ed.). (1995). Issues and methods in Rorschach research.Hillsdale, NJ: Erlbaum.

Exner, J. E. (1999). The Rorschach: Measurement concepts and issues ofvalidity. In S. E. Embretson & S. L. Hershberger (Eds.), The new rulesof measurement: What every psychologist and educator should know(pp. 159-183). Mahwah, NJ: Erlbaum.

Exner, J. E. (2000). A primer for Rorschach interpretation. Asheville, NC:Rorschach Workshops.

Exner, J. E., Armbruster, G. L., & Viglione, D. (1978). The temporalstability of some Rorschach features. Journal of Personality Assess-ment, 42, 474-482.

Exner, J. E., Colligan, S. C., Hillman, L. B., Metis, A. S., Ritzier, B.,Rogers, K. T., Sciara, A., D., & Viglione, D. J. (2001). A Rorschachworkbook for the Comprehensive System (5th ed.). Asheville, NC: Ror-schach Workshops.

Exner, J. E., Colligan, S. C., Hillman, L. B., Ritzier, B., Sciara, A. D., &Viglione, D. J. (1995). A Rorschach workbook for the ComprehensiveSystem (4th ed.). Asheville, NC: Rorschach Workshops.

Exner, J. E., Thomas, E. A., & Mason, B. J. (1985). Children's Rorschachs:Description and prediction. Journal of Personality Assessment, 49, 13-20.

Exner, J. E., & Weiner, I. B. (1982). The Rorschach: A ComprehensiveSystem: Vol. 3. Assessment of children and adolescents. New York:Wiley.

Exner, J. E., & Weiner, I. B. (1995). The Rorschach: A ComprehensiveSystem: Vol. 3. Assessment of children and adolescents (2nd ed.). NewYork: Wiley.

Exner, J. E., Weiner, I. B., & Schuyler, W. (1976). A Rorschach workbookfor the Comprehensive System. Bayville, NY: Rorschach Workshops.

Exner, J. E., & Wylie, J. (1977). Some Rohrschach data concerningsuicide. Journal of Personality Assessment, 41, 339-348.

Eyman, J. R., & Eyman, S. K. (1991). Personality assessment in suicideprediction. Special Issue: Assessment and prediction of suicide. Suicideand Life Threatening Behavior, 21, 37-55.

Eyman, J. R., & Eyman, S. K. (1992). Personality assessment in suicideprediction. In R. W. Maris, A. L. Berman, J. T. Maltsberger, & R. I.

Yufit (Eds.), Assessment and prediction of suicide (pp. 183—201). NewYork: Guilford Press.

Eyman, S. K., & Eyman, J. R. (1987, August-September). An investigationof Exner's Suicide Constellation. Paper presented at the 95th AnnualConvention of the American Psychological Association, New York.

Finn, S. E., & Kamphuis, i. H. (1995). What a clinician needs to knowabout baserates. In J. N. Butcher (Ed.), Clinical personality assessment(pp. 224-235). New York: Oxford University Press.

Rske, D. W., Cartwright, D. S., & Kirtner, W. L. (1964). Are psychother-apeutic changes predictable? Journal of Abnormal and Social Psychol-ogy, 69, 413-426.

Fowler, C., Hilsenroth, M. J., & Handler, L. (1996). A multimethodapproach to assessing dependency: The early memory dependencyprobe. Journal of Personality Assessment, 67, 399-413.

Fowler, C. J., Piers, C. C., Hilsenroth, M. J., Holdwick, D., & Padawer, R.(2001). Assessing risk factors various degrees of suicidal activity: TheRorschach Suicide Constellation. Journal of Personality Assessment, 76,333-351.

Franklin, K. W., & Cornell, D. G. (1997). Rorschach interpretation withhigh-ability adolescent females: Psychopathology or creative thinking?Journal of Personality Assessment, 68, 184-196.

Frueh, B. C., & Kinder, B. N. (1994). The susceptibility of the Rorschachinkblot test to malingering of combat-related PTSD. Journal of Person-ality Assessment, 62, 280-298.

Gacono, C. B., & Meloy, J. R. (1994). The Rorschach assessment ofaggressive and psychopathic personalities. Mahwah, NJ: Erlbaum.

Gadol, I. (1969). The incremental and predictive validity of the Rorschachtest in personality assessment of normal, neurotic, and psychotic sub-jects. Dissertation Abstracts International, 29, 3482-B. (University Mi-crofilms No. 3469-4469)

Gallucci, N. T. (1989). Personality assessment with children of superiorintelligence: Divergence versus psychopathology. Journal of Personal-ity Assessment, 53, 749-760.

Garb, H. N. (1989). Clinical judgment, clinical training, and professionalexperience. Psychological Bulletin, 105, 387-396.

Garb, H. N. (1994). Cognitive heuristics and biases in personality assess-ment. In L. Heath, R. S. Tindale, J. Edwards, E. J. Posavac, F. B. Bryant,E. Henderson-King, Y. Suarez-Balcazar, & J. Myers (Eds.), Applicationsof heuristics and biases to social issues. New York: Plenum Press.

Garb, H. N. (1998). Studying the clinician: Judgment research and psy-chological assessment. Washington, DC: American PsychologicalAssociation.

Garb, H. N. (1999). Call for a moratorium on the use of the RorschachInkblot Test in clinical and forensic settings. Assessment, 6, 313-315.

Garb, H. N., Florio, C. M., & Grove, W. M. (1998). The validity of theRorschach and the Minnesota Mulu'phasic Personality Inventory: Re-sults from meta-analyses. Psychological Science, 9, 402-404.

Garb, H. N., Wood, J. M., Nezworski, M. T., Grove, W. M., & Stejskal,W. J. (2001). Toward a resolution of the Rorschach controversy. Psy-chological Assessment, 13, 433-448.

Garfield, S. L. (2000). The Rorschach test in clinical diagnosis—A briefcommentary. Journal of Clinical Psychology, 56, 431-434.

Golden, M. (1964). Some effects of combining psychological tests onclinical inferences. Journal of Consulting Psychology, 28, 440-446.

Greenwald, D. F. (1991). Personality dimensions reflected by the Ror-schach and the 16PF. Journal of Clinical Psychology, 47, 708-715.

Haller, N., & Exner, J. E. (1985). The reliability of Rorschach variables forinpatients presenting symptoms of depression and/or helplessness. Jour-nal of Personality Assessment, 49, 516-521.

Hamel, M., Shaffer, T. W., & Erdberg, P. (2000). A study of nonpatientpreadolescent Rorschach protocols. Journal of Personality Assess-ment, 75, 280-294.

Hammond, K. R. (1955). Probabilistic functioning and the clinical method.Psychological Review, 62, 255-262.


Hathaway, S. R., McKinley, J. C, Butcher, J. N., Dahlstrom, W. G.,Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota Mul-tiphasic Personality Inventory-2: Manual for administration and scor-ing. Minneapolis: University of Minnesota Press.

Miller, J. B., Rosenthal, R., Bornstein, R. F., & Berry, D. T. (1999). Acomparative meta-analysis of Rorschach and MMPI validity. Psycho-logical Assessment, 11, 278-296.

Hilsenroth, M. J., Fowler, C. J., & Padawer, J. R. (1998). The RorschachSchizophrenia Index (SCZI): An examination of reliability, validity, anddiagnostic efficiency. Journal of Personality Assessment, 70, 514-534.

Hilsenroth, M. J., Fowler, J. C., Padawer, J. R., & Handler, L. (1997).Narcissism in the Rorschach revisited: Some reflections on empiricaldata. Psychological Assessment, 9, 113-121.

Hilsenroth, M. J., & Handler, L. (1995). A survey of graduate students'experiences, interests, and attitudes about learning the Rorschach. Jour-nal of Personality Assessment, 64, 243—257.

Hilsenroth, M. J., Handler, L., Toman, K. M., & Padawar, J. R. (1995).Rorschach and MMPI-2 indices of early psychotherapy termination.Journal of Consulting and Clinical Psychology, 63, 956—965.

Holaday, M., & Whittenberg, T. (1994). Rorschach responding in childrenand adolescents who have been severely burned. Journal of PersonalityAssessment, 62, 269-279.

Holzman, P. S., et al. (1974). Eye-tracking dysfunctions in schizophrenicpatients and their relatives. Archives of General Psychiatry, 31, 143-151.

Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach:Unfulfilled promises and an uncertain future. Psychological Assess-ment. 11, 266-277.

Hunsley, J., Hanson, R. K., & Parker, K. C. (1988). A summary of thereliability and stability of MMPI scales. Journal of Clinical Psychol-ogy, 44, 44—46.

Johnston, M. H., & Holzman, P. S. (1979). Assessing schizophrenic think-ing: A clinical and research instrument for measuring thought disorder.San Francisco: Jossey-Bass.

Kates, J. (1994). A validation study of the Rorschach Ego ImpairmentIndex. Unpublished doctoral dissertation, California School of Profes-sional Psychology, San Diego, CA.

Kleiger, J. H. (1999). Disordered thinking and the Rorschach: Theory,research, and differential diagnosis. Hillsdale, NJ: Analytic Press.

Kleinmuntz, B. (1990). Why we still use our heads instead of formulas:Toward an integrative approach. Psychological Bulletin, 107, 296-310.

Krall, V., Sachs, H., Langer, B., Rayson, B., & Growe, G. (1983). Ror-schach norms for inner city children. Journal of Personality Assess-ment, 47, 155-157.

Leichtman, M. (1996). The Rorschach: A developmental perspective. Hills-dale, NJ: Analytic Press.

Lerner, H. D., & St. Peter, S. (1984a). Patterns of object relations inneurotic, borderline and schizophrenic patients. Psychiatry, 47, 77-92.

Lemer, H. D., & St. Peter, S. (1984b). The Rorschach H response andobject relations. Journal of Personality Assessment, 48, 345-350.

Levy, L. H. (1963). Psychological interpretation. New York: Holt, Rine-hart & Winston.

Masling, J. M. (1986). Empirical studies of psychoanalytic theories. Hills-dale, NJ: Erlbaum.

Masling, J. M., Rabid, L., & Blondheim, S. H. (1967). Obesity, level ofaspiration, and Rorschach and TAT measures of oral dependence. Jour-nal of Consulting Psychology, 31, 233-239.

Mattlar, C. E., Carlsson, A., & Forsander, C. (1993). The issue of thepopular response: Definition and universal vs. culture-specific popularresponses. Special Issue: International Rorschach Congress 1993. BritishJournal ofProjective Psychology, 38, 53-62.

Mauger, P. A. (1972). The test-retest reliability of persons: An empir-ical investigation utilizing the MMPI and the Personality Research

Form. Unpublished doctoral dissertation, University of Minnesota,Minneapolis.

McCann, J. T. (1998). Defending the Rorschach in court: An analysis ofadmissibility using legal and professional standards. Journal of Person-ality Assessment, 70, 125-144.

McDowell, C. J., & Acklin, M. W. (1996). Standardizing procedures forcalculating Rorschach interrater reliability: Conceptual and empiricalfoundations. Journal of Personality Assessment, 66, 308-320.

Meehl, P. E. (1957). When shall we use our heads instead of the formula?Journal of Counseling Psychology, 4, 268-273.

Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the effi-ciency of psychometric signs, patterns, or cutting scores. PsychologicalBulletin, 52, 171-190.

Meloy, J. R., Hansen, T. L., & Weiner, I. B. (1997). Authority of theRorschach: Legal citations during the past 50 years. Journal of Person-ality Assessment, 69, 53—62.

Meyer, G. J. (1993). The impact of response frequency on the Rorschachconstellation indices and on their validity with diagnostic and MMPI-2criteria. Journal of Personality Assessment, 60, 153-180.

Meyer, G. J. (1997a). Assessing reliability: Critical correlations for acritical examination of the Rorschach Comprehensive System. Psycho-logical Assessment, 9, 480-489.

Meyer, G. J. (1997b). Thinking clearly about reliability: More criticalcorrelations regarding the Rorschach Comprehensive System. Psycho-logical Assessment, 9. 495-498.

Meyer, G. J. (1999). Introduction to the Special Series on the utility of theRorschach for clinical assessment. Psychological Assessment, 11, 235-239.

Meyer, G. J. (2000). Incremental validity of the Rorschach PrognosticRating Scale over the MMPI Ego Strength Scale and IQ. Journal ofPersonality Assessment, 74, 356-370.

Meyer, G. J. (2001). Evidence to correct misperceptions about Rorschachnorms. Manuscript submitted for publication.

Meyer, G. J. (in press). Exploring possible ethnic differences and bias inRorschach Comprehensive System scores. Journal of Personality As-sessment.

Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Kubiszyn, T. W.,Moreland, K. L., Eisman, E. J., & Dies, R. R. (1998). Benefits and costsof psychological assessment in healthcare delivery: Report of the Boardof Professional Affairs Psychological Assessment Work Group, Part 1.Washington, DC: American Psychological Association.

Meyer, G. J., & Handler, L. (1997). The ability of the Rorschach to predictsubsequent outcome: A meta-analysis of the Rorschach Prognostic Rat-ing Scale. Journal of Personality Assessment, 69, 1-38.

Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E., Jr., Fowler, C. J.,Piers, C. C., & Resnick, J. (in press). An examination of interraterreliability for scoring the Rorschach Comprehensive System in eightdata sets. Journal of Personality Assessment.

Meyer, G. J., & Resnick, G. D. (1996, February). Assessing ego impair-ment: Do scoring procedures make a difference? Paper presented at the15th International Research Conference, Boston.

Milott, S. R., Lira, F. T., & Miller, W. C. (1977). Psychological assessmentof the burned patient. Journal of Clinical Psychology, 33, 425-430.

Netter, B. E. C., & Viglione, D. J. (1994). An empirical study of malin-gering schizophrenia on the Rorschach. Journal of Personality Assess-ment, 62, 45-57.

O'Connell, M., Cooper, S., Perry, C., & Hoke, L. (1989). The relationshipbetween thought disorder and psychotic symptoms in borderline person-ality disorder. Journal of Nervous and Mental Disease, 177, 273-278.

Paolo, A. M., Troster, A. I., & Ryan, J. J. (1997). Test-retest stability of theCalifornia Learning Test in older persons. Neuropsychology, 11, 613-616.

Parker, K. C., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorschach, and


WAIS: A meta-analytic comparison of reliability, stability, and validity.Psychological Bulletin, 103, 367-373.

Parker, K. C. H., Hunsley, J., & Hanson, R. K. (1999). Old wine from oldskins sometimes taste like vinegar: A response to Garb, Florio, andGrove. Psychological Science, 10, 367-373.

Perez, F. I. (1976). Behavioral analysis of clinical judgment. Perceptualand Motor Skills, 43, 711-718.

Perry, W., & Braff, D. L. (1994). Information-processing deficits andthought disorder in schizophrenia. American Journal of Psychiatry, 151,363-367.

Perry, W., & Braff, D. L. (in press). The simultaneous measurement ofattention and information processing and disturbed thinking in maleschizophrenic patients. Archives of General Psychiatry.

Perry, W., Geyer, M. A., & Braff, D. L. (1999). Sensorimotor gating andthought disturtence measured in close temporal proximity in schizo-phrenic patients. Archives of General Psychiatry, 56, 277—281.

Perry, W., Moore, D., & Braff, D. L. (1995). Gender differences on thoughtdisturbance measures in schizophrenia. American Journal of Psychiatry,152, 1298-1301.

Perry, W., Potterat, E. G., Auslander, L., Kaplan, E., & Jeste, D. (19%). Aneuropsychological approach to the Rorschach in patients with dementiaof the Alzheimer type. Assessment, 3, 351-363.

Perry, W., Sprock, J., Schaible, D., McDougall, A., et al. (1995). Amphet-amine on Rorschach measures in normal subjects. Journal of PersonalityAssessment, 64, 456-465.

Perry, W., & Viglione, D. J. (1991). The Ego Impairment Index as apredictor of outcome in melancholic depressed patients treated withtricyclic antidepressants. Journal of Personality Assessment, 56, 487-501.

Perry, W., Viglione, D., & Braff, D. L. (1992). The Ego Impairment Indexand schizophrenia: A validation study. Journal of Personality Assess-ment, 59, 165-175.

Peterson, C., Samel, A., von Baeyer, C., Abramson, L. H., Metalsky, G. I.,& Seligman, M. E. P. (1982). The Attributional Style Quesionnaire.Cognitive Theory and Research, 6, 289-299.

Piotrowski, C., & Belter, R. W. (1999). Internship training in psychologicalassessment: Has managed care had an impact? Assessment, 6, 381-389.

Presley, G., Smith, C., Hilsenroth, M., & Exner, J. (in press). Rorschachvalidity with African Americans. Journal of Personality Assessment.

The Psychological Corporation. (1997). Wechsler Adult Intelligence Scale,technical manual (3rd ed.). San Antonio, TX: The PsychologicalCorporation.

Putnam, S. H., Kurtz, J. E., & Houts, D. C. (1996). Four-month test-retestreliability of the MMPI-2 with normal male clergy. Journal of Person-ality Assessment, 67, 341-353.

Rapaport, D., Gill, M., & Schafer, R. (1946). Diagnostic psychologicaltesting (Vols. 1 & 2). Chicago: Yearbook Publishers.

Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistencyof personality traits from childhood to old age: A quantitative review oflongitudinal studies. Psychological Bulletin, 126, 3—25.

Rogers, R. (Ed.). (1997). Clinical assessment of malingering and deception(2nd ed.). New York: Guilford Press.

Russ, S. W. (1980). Primary process integration on the Rorschach andachievement in children. Journal of Personality Assessment, 44, 338-344.

Russ, S. W. (1981). Primary process integration on the Rorschach andachievement in children: A follow-up study. Journal of PersonalityAssessment, 45, 473-477.

Ryan, J. J., Dunn, G. E., & Paolo, A. M. (1995). Temporal stability of theMMPI-2 in a substance abuse sample. Psychotherapy in Private Prac-tice, 14, 33-41.

Schafer, R. (1954). Psychoanalytic interpretation in Rorschach testing.New York: Grune & Stratton.

Schaich, D. (1999). A synthesis of the conceptual and inferential proce-

dures involved with the assessment of malingering: An applied study.Unpublished doctoral dissertation, California School of ProfessionalPsychology, San Diego, CA.

Schuerger, J. M., Zarrella, K. L., & Holtz, A. S. (1989). Factors thatinfluence the temporal stability of personality by questionnaire. Journalof Personality and Social Psychology, 56, 777-783.

Schwartz, N. S., Mebane, D. L., & Malony, H. N. (1990). Effects ofalternate modes of administration of Rorschach performance of deafadults. Journal of Personality Assessment, 54, 671-683.

Shaffer, T. W., Erdberg, P., & Haroian, J. (1999). Current nonpatient datafor the Rorschach, WAIS-R, and MMPI-2. Journal of PersonalityAssessment, 73, 305-316.

Shapiro, J. P., Leifer, M., Martone, M. W., & Kassem, L. (1990). Multi-method assessment of depression in sexually abused girls. Journal ofPersonality Assessment, 55, 234-248.

Shenton, M. E., Solovay, M. R., et al. (1989). Thought disorder in therelatives of psychotic patients. Archives of General Psychiatry, 46,897-901.

Shrout, P. E., & Fliess, J. L. (1979). Intraclass correlations: Uses inassessing rater reliability. Psychological Bulletin, 86, 420-428.

Silberg, J. L., & Armstrong, J. G. (1992). The Rorschach test for predictingsuicide among depressed adolescent inpatients. Journal of PersonalityAssessment, 59, 290-303.

Sines, L. K., Silver, R. J., & Lucero, R. J. (1961). The effect of therapeuticintervention by untrained "therapists." Journal of Clinical Psychol-ogy, 17, 394-396.

Skelton, M. D., Boik, R. ;., & Madero, J. N. (1995). Thought disorder onthe WAIS-R relative to the Rorschach: Assessing identity disorderadolescents. Journal of Personality Assessment, 65, 533-549.

Sloan, P., Arsenault, L., Hilsenroth, M., Handler, L., & Harvill, L. (1996).Rorschach measures of posttraumatic stress in Persian Gulf War veter-ans. Journal of Personality Assessment, 66, 54-64.

Sloan, P., Arsenault, L., Hilsenroth, M., Harvill, L., & Handler, L. (1995).Rorschach measures of posttraumatic stress in Persian Gulf War veter-ans. Journal of Personality Assessment, 64, 397-414.

Spiro, A., Butcher, J. N., Levenson, M. R., Aldwin, C. M., & Bosse, R.(1993, August). Personality change and stability over five years: TheMMPI-2 in older men. Paper presented at the 101st Annual Conventionof the American Psychological Association, Toronto, Ontario, Canada.

Stedman, J., Hatch, J., & Schoenfeld, L. (2000). Preintemship preparationin psychological testing and psychotherapy: What internship directorssay they expect. Professional Psychology: Research and Practice, 31,321-326.

Stone, L. A. (1965). Test-retest stability of the MMPI scales. Psycholog-ical Reports, 16, 619-620.

Strieker, G., & Gold, J. R. (1999). The Rorschach: Toward a nomotheti-cally based, idiographically applicable configurational model. Psycho-logical Assessment, 11, 240-250.

Turner, D. R. (1966). Predictive efficiency as a function of amount ofinformation and level of professional experience. Journal of ProjectiveTechniques and Personality Assessment, 30, 4-11. .

Urist, J. (1977). The Rorschach Test and the assessment of object relations.Journal of Personality Assessment, 41, 3-9.

Viglione, D. J. (1989). Rorschach science and art. (Review of the Ror-schach: A Comprehensive System, Vol. 1, 2nd ed.). Journal of Person-ality Assessment, 53, 195-197.

Viglione, D. J. (1997). Problems in Rorschach research and what to doabout them. Journal of Personality Assessment, 68, 590-599.

Viglione, D. J. (1999). A review of recent research addressing the utility ofthe Rorschach. Psychological Assessment, 11, 251-265.

Viglione, D. J. (2001). Challenges and solutions in Rorschach coding andadministration. Manuscript in preparation.

Viglione, D. J., & Exner, J. E. (1983). The effects of state anxiety and


limited social-evaluative stress on the Rorschach. Journal of PersonalityAssessment, 47, 150-154.

Viglione, D. J., Gaudiana, S., & Gowri, A. (1997, April). Two questionsabout Rorschach norms: Cultural issues and divergence. Paper pre-sented at the Society for Personality Assessment, San Diego, CA.

Wade, T. C., & Baker, T. B. (1977). Opinions and use of psychologicaltests: A survey of clinical psychologists. American Psychologist, 32,874-882.

Wahlberg, K. E., Wynne, L. C., Oja, H., Keskitalo, P., Pykalainen, L.,Lahti, I., Moring, J., Naarala, M., Sorri, A., Seitamaa, M., Laksy, K.,Kolassa, J., & Tienari, P. (1997). Gene-environment interaction invulnerability to schizophrenia: Findings from the Finnish adoptive fam-ily study of schizophrenia. American Journal of Psychiatry, 154, 355-362.

Watkins, C., Campbell, V., Nieberding, R., & Hallmark, R. (1995). Con-temporary practice of psychological assessment by clinical psycholo-gists. Professional Psychology: Research and Practice, 26, 54—60.

Weiner, I. B. (1966). Psychodiagnosis in schizophrenia. New York: Wiley.Weiner, I. B. (1996). Some observations on the validity of the Rorschach

Inkblot Method. Psychological Assessment, 8, 206-213.Weiner, I. B. (1998). Principles of Rorschach interpretation. Mahwah, NJ:

Erlbaum.Weiner, I. B. (1999). What the Rorschach can do for you: Incremental

validity in clinical applications. Assessment, 6, 327-338.Weiner, I. B. (2001a). Advancing the science of psychological assessment:

The Rorschach Inkblot Method as exemplar. Psychological Assess-ment, 13, 423-432.

Weiner, I. B. (2001b). Considerations in correcting Rorschach referencedate. Journal of Personality Assessment, 77, 122—127.

Weiner, I. B., Exner, J. E., Jr., & Scbiara, A. (1996). Is the Rorschachwelcome in the courtroom? Journal of Personality Assessment, 67,422-424.

Wood, J. M., & Lilienfeld, S. O. (1999). The Rorschach Inkblot Test: Acase of overstatement? Assessment, 6, 341-349.

Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000).The Rorschach test in clinical diagnosis: A critical review with abackward look at Garfield (1947). Journal of Clinical Psychology, 56,395-430.

Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996a). The Compre-hensive System for the Rorschach: A critical examination. PsychologicalScience, 7, 3-10.

Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996b). Thinkingcritically about the Comprehensive System for the Rorschach: A reply toExner. Psychological Science, 7, 14-17.

Wood, J. M., Nezworski, M. T., Stejskal, W. J., Garven, S., & West, S. G.(1999). Methodological issues in evaluating Rorschach validity: A com-ment on Burns and Viglione (1996), Weiner (19%), and Ganellen(1996). Assessment, 6, 115-129.

Received July 27, 2000Revision received February 26, 2001

Accepted July 3, 2001

Low Publication Prices for APA Members and Affiliates

Keeping you up-to-date. All APA Fellows, Members, Associates, and Student Affiliatesreceive—as part of their annual dues—subscriptions to the American Psychologist andAPA Monitor. High School Teacher and International Affiliates receive subscriptions tothe APA Monitor, and they may subscribe to the American Psychologist at a significantlyreduced rate. In addition, all Members and Student Affiliates are eligible for savings of upto 60% (plus a journal credit) on all other APA journals, as well as significant discounts onsubscriptions from cooperating societies and publishers (e.g., the American Association forCounseling and Development, Academic Press, and Human Sciences Press).

Essential resources. APA members and affiliates receive special rates for purchases ofAPA books, including the Publication Manual of the American Psychological Association,and on dozens of new topical books each year.

Other benefits of membership. Membership in APA also provides eligibility forcompetitive insurance plans, continuing education programs, reduced APA convention fees,and specialty divisions.

More information. Write to American Psychological Association, Membership Services,750 First Street, NE, Washington, DC 20002-4242.

the rorschach: facts, fictions, and...

Documents