Assessment20(4) 448 –461© The Author(s) 2013Reprints and permissions: sagepub.com/journalsPermissions.navDOI: 10.1177/1073191113478154asm.sagepub.com

It has long been recognized that the validity of self-report–based personality assessment is limited by the degree to which the individual assessed accurately communicates his or her inner experiences and perceptions. Whether this communication occurs via interview or psychological test-ing, issues such as limited insight or purposeful distortion can lead to inaccurate results on personality inventories. Unlike interview data, which rely on the interviewer’s sub-jective assessment of the veracity of the interviewee’s self-report, in personality-based psychological testing, two potentially biasing response styles, Noncontent-Based and Content-Based Invalid Responding (CBIR), can be assessed through empirically validated objective means (i.e., scales designed to measure such response styles; Ben-Porath, 2013). Individuals who engage in Noncontent-Based Invalid Responding (NCIR) fail to respond or provide ran-dom or fixed responses to test items regardless of their con-tent. CBIR occurs when the test taker portrays him or herself as functioning better (under-reporting) or worse (over-reporting) than would be indicated by an objective assessment of his or her functioning.

Given the potential costs of misinterpreting psychologi-cal test results (e.g., in outpatient and/or inpatient settings where an individual is prescribed medication or forensic

settings where an individual is deemed incompetent to pro-ceed to trial because of psychological difficulties), once NCIR has been ruled out, being able to effectively detect CBIR is of paramount importance. Three questions need to be considered when selecting a psychological test and con-sidering its ability to detect CBIR. First, does the instrument include scales designed to assess CBIR? If so, have these scales been empirically validated? Finally, does any CBIR they detect generalize to other psychological tests taken at or near the same time?

With regard to the first two questions just listed, the Minnesota Multiphasic Personality Inventory–2–Restructured Form (MMPI-2-RF; Ben-Porath & Tellegen, 2008; Tellegen &

478154 ASM20410.1177/1073191113478154AssessmentForbey et al.© The Author(s) 2011

Reprints and permission: http://www.sagepub.com/journalsPermissions.nav

1Ball State University, Muncie, IN, USA2Kent State University, Kent, OH, USA3Minneapolis Veterans Administration Medical Center, Minneapolis, MN, USA4University of Minnesota, Minneapolis, MN, USA5Ann Arbor, MI, USA

Corresponding Author:Johnathan D. Forbey, Department of Psychological Science, Ball State University, North Quad, Muncie, IN 47306, USA.Email: jdforbey@bsu.edu

Associations Between MMPI-2-RF Validity Scale Scores and Extra-Test Measures of Personality and Psychopathology

Johnathan D. Forbey1, Tayla T. C. Lee2, Yossef S. Ben-Porath2, Paul A. Arbisi3,4, and Diane Gartland5

Abstract

The current study explored associations between two potentially invalidating self-report styles detected by the Validity scales of the Minnesota Multiphasic Personality Inventory–2–Restructured Form (MMPI-2-RF), over-reporting and under-reporting, and scores on the MMPI-2-RF substantive, as well as eight collateral self-report measures administered either at the same time or within 1 to 10 days of MMPI-2-RF administration. Analyses were conducted with data provided by college students, male prisoners, and male psychiatric outpatients from a Veterans Administration facility. Results indicated that if either an over- or under-reporting response style was suggested by the MMPI-2-RF Validity scales, scores on the majority of the MMPI-2-RF substantive scales, as well as a number of collateral measures, were significantly affected in all three groups in the expected directions. Test takers who were identified as potentially engaging in an over- or under-reporting response style by the MMPI-2-RF Validity scales appeared to approach extra-test measures similarly regardless of when these measures were administered in relation to the MMPI-2-RF. Limitations and suggestions for future study are discussed.

Keywords

personality, assessment, validity, interpretability, college, outpatients, veterans

Article

Forbey et al. 449

Ben-Porath, 2008/2011) contains well-validated measures of both NCIR and CBIR (Ben-Porath, 2013). In terms of NCIR, the MMPI-2-RF retains the use of the Cannot Say Scale (CNS) Index, which is a count of omitted or double-answered items. The MMPI-2-RF also includes restructured versions of the MMPI-2 Variable Response Inconsistency (VRIN) and True Response Inconsistency (TRIN) scales, VRIN-r and TRIN-r, which assess random and fixed responding, respec-tively. In terms of CBIR, the MMPI-2-RF contains five pri-mary scales designed to detect over-reporting. Three of the CBIR over-reporting indicators are revised versions of MMPI-2 scales, including Infrequent Responses (F-r), Infrequent Psychopathology Responses (F

P-r), and Symptom

Validity (FBS-r). Infrequent Somatic Responses (FS) and the

Response Bias Scale (RBS) were added to the MMPI-2-RF to assess the report of unusual somatic symptoms and non-credible memory complaints, respectively. The MMPI-2-RF also contains two revised MMPI-2 scales designed to detect under-reporting, Uncommon Virtues (L-r), and Adjustment Validity (K-r).

A number of studies have supported the ability of the over-reporting scales of the MMPI-2-RF to detect invalid response styles. For example, two studies (Marion, Sellbom, & Bagby, 2011; Sellbom & Bagby, 2010) reported that the MMPI-2-RF over-reporting scales were able to dif-ferentiate between college students coached to feign men-tal illness and psychiatric patients. Furthermore, Wygant et al. (2011) reported that the MMPI-2-RF over-reporting scales were able to accurately identify individuals under-going compensation-seeking evaluations who were classi-fied as probable/definite malingerers based on structured criteria for malingered neurocognitive dysfunction (Slick, Hopp, Strauss, & Thompson, 1997) or malingered pain-related disability (Bianchini, Greve, & Glynn, 2005). All the studies just mentioned demonstrated a large effect size for the ability of the MMPI-2-RF CBIR over-reporting scales in differentiating between groups.

With respect to under-reporting, Sellbom and Bagby (2008) reported that L-r and K-r were able to differentiate between both college students and psychiatric patients instructed to under-report problems and individuals instructed to take the test under standard instructions. Sellbom and Bagby (2008) concluded that their results support the utility of the MMPI-2-RF under-reporting indicators and noted that the effect sizes they obtained were similar to the large effect sizes reported in Baer and Miller’s (2002) meta-analysis of the MMPI-2 measures of under-reporting, from which L-r and K-r were derived. Data reported in the MMPI-2-RF Technical Manual indicate that scores on these scales are commonly elevated in samples of individuals tested under circumstances that are likely to motivate under-reporting (e.g., personnel screening and child custody litigation).

With respect to the third question listed earlier concern-ing the generalizability of MMPI-2-RF CBIR findings to

other measures, only limited MMPI-2 research has been conducted to date. Garcia, Franklin, and Chambliss (2010) examined whether invalid response styles detected by the MMPI-2 Validity scales were correlated with both MMPI-2 substantive scale scores and conjointly administered col-lateral measures in a sample of veterans diagnosed with posttraumatic stress disorder (PTSD). Garcia et al. (2010) demonstrated that scores on MMPI-2 Clinical scales and conjointly administered measures of depression and PTSD symptoms were significantly higher for individuals with elevated scores on the F (Infrequency) scale. Although this study had some methodological shortcomings that may have artificially increased the detected over-reporting rate (i.e., using nonstandard cut scores for some validity scales), overall, results indicated that MMPI-2-detected over-reporting generalized to the other conjointly administered measures.

Forbey and Lee (2011) attempted to further clarify the association between under- and over-reporting response styles scores on the MMPI-2 and conjointly administered measures of social, behavioral, and psychological difficul-ties. The authors used data provided by 1,112 undergradu-ates who took the MMPI-2 and a number of collateral measures together as part of a larger study. After excluding NCIR individuals, three groups of participants were identi-fied based on their Validity scale scores: CBIR over-report-ers (CBIR-OR), CBIR under-reporters (CBIR-UR), and “within normal limits” (WNL) responders on the validity scales. In terms of MMPI-2 substantive scales, results mir-rored those of previous coaching and simulation research, which has long established that scores on MMPI-2 sub-stantive scales vary substantially as a function of validity scale scores. More important, with respect to the conjointly administered collateral measures, Forbey and Lee (2011) found 27 of 35 collateral measures’ total and/or subscale scores in the CBIR-OR group were significantly different from the WNL responders (with small to large median effect sizes), and 15 of 35 collateral total scale and/or sub-scale scores were significantly different in the CBIR-UR group when compared with the WNL responders (with medium to large median effect sizes). Overall, these results indicated that individuals who were identified as poten-tially engaging in CBIR-OR by the MMPI-2 Validity scales reported significantly higher levels of psychopathology on the MMPI-2 substantive scales and on conjointly adminis-tered collateral measures when compared with individuals with WNL validity scores. Individuals identified as poten-tially engaging in CBIR-UR by the MMPI-2 Validity scales generally reported substantially lower (than the WNL group) levels of psychopathology on both the MMPI-2 substantive scales and collateral measures administered at the same time.

The study by Forbey and Lee (2011) indicated that MMPI-2 detected CBIR-OR and CBIR-UR likely reflects a

450 Assessment 20(4)

more generalized response style that affects conjointly administered collateral measures. This finding is of critical importance because the MMPI-2 or MMPI-2-RF are often administered as part of a battery that includes other tests that lack empirically validated CBIR measures. Being able to rely on MMPI Validity scale findings to guide interpreta-tion of these additional measures may increase their utility considerably. However, Forbey and Lee’s (2011) study was limited in generalizability, as it only included college stu-dents who were administered all the measures conjointly (i.e., on the same occasion). Furthermore, this study used the MMPI-2, rather than the MMPI-2-RF, though use of the MMPI-2-RF may confer some advantages, including the ability to examine additional over-reporting scales and the reduced item overlap among the validity scales (in particu-lar the removal of the item overlap among the under- and over-reporting scales that existed on the MMPI-2). Moreover, because it is approximately 40% shorter than the MMPI-2, the MMPI-2-RF is more likely to be administered along with other self-report measures.

The current study was conducted to build on Forbey and Lee’s (2011) findings. The goals of the study were to examine associations between CBIR, as detected by MMPI-2-RF Validity scales and scores on the substantive scales of the test (to serve as an overall procedural evalua-tion of the methodology) and, more important, to examine associations between the CBIR indicators and conjointly administered self-report measures (which lack such valid-ity indicators) in different populations and during varying time frames. Specifically, using a criterion groups design, we examined MMPI-2-RF and criterion measure scale score differences between subgroups of individuals who were classified as engaging in CBIR-OR or CBIR-UR and individuals who produced interpretable (i.e., WNL on all the validity scales) MMPI-2-RF protocols in male and female college students, males undergoing intake to a cor-rectional facility, and male Veterans Administration men-tal health outpatients.

Based on previous research, we hypothesized that when compared with individuals classified as responding can-didly, individuals engaging in an CBIR-OR or CBIR-UR would score differently on MMPI-2-RF substantive scales and collateral measures administered conjointly or within a specified amount of time. Specifically, we hypothesized that individuals identified as potentially engaging in CBIR-OR based on MMPI-2-RF Validity scale cut scores (i.e., elevations on F-r, F

P-r, FS, FBS-r, and/or RBS) would

score higher than those assigned to the WNL group on substantive measures. We further hypothesized that indi-viduals who were identified as potentially engaging in CBIR-UR based on their cut -scores on MMPI-2-RF-specific Validity scales (i.e., elevations on L-r and/or K-r) would score lower than those on the WNL group on sub-stantive measures.

MethodParticipantsThree archival samples of participants were used in the cur-rent study, with data originally being collected from these participants as part of a larger series of unpublished studies examining the comparability between conventional and several modularized1 versions of the MMPI-2–computerized adaptive version (MMPI-2-CA; Forbey & Ben-Porath, 2007). The first sample, composed of college participants, included 1,194 (487 men and 707 women) undergraduate students from a Midwestern U.S. university enrolled in Introductory Psychology classes. The second sample, made up of correctional participants, included 632 male inmates from a large Midwestern U.S. intake correctional facility who volunteered to participate in the original study. The third sample included 181 men who were recruited from outpatient psychiatric care Veterans Affairs facility at a large tertiary care medical center.

To reduce error variance in our analyses, individuals in each of the samples who provided NCIR MMPI-2-RF profiles, defined by CNS scores ≥15 and/or T scores ≥80 on VRIN-r or TRIN-r based on the recommendations in the MMPI-2-RF administration manual (Ben-Porath & Tellegen, 2008), were removed from the current study. This resulted in a total of 129 (10.8%) college, 19 (3.0%) cor-rectional, and 17 (9.4%) psychiatric participants being excluded from analyses. Among the college participants, no statistically significant differences between included and excluded participants were evident in terms of age or ethnicity/racial group. However, a statistically significant differences of a small effect size (Cohen, 1988) was evident for gender, χ2(1) = 26.986, p < .001, Φ = .15) with men being slightly more likely to produce NCIR MMPI-2-RF profiles. Among the correctional and psychiatric participants, no sta-tistically significant differences between included and excluded were evident in terms of age. However, a statisti-cally significant difference of a small effect size (Cohen, 1988) was evident for ethnicity/racial group in both sets of participants, χ2(2) = 8.526, p < .05, Φ = .12 and χ2(2) = 14.474, p < .001, Φ = .28, respectively, with minority par-ticipants being slightly more likely to produce NCIR MMPI-2-RF profiles.

After NCIR exclusions, the final sample of 1,065 college participants included 407 men and 658 women (age range = 18-48 years, M = 19.60 years, SD = 3.14) who were Caucasian (n = 947, 88.9%), African American (n = 84, 7.9%), or of another or unidentified ethnicity (n = 34, 3.2%). The final sample of correctional participants included 613 men (age range = 18-66 years, M = 32.37 years, SD = 9.94) who were Caucasian (n = 310, 50.6%), African American (n = 173, 28.2%), or of another or unidentified ethnicity (n = 130, 21.2%). The final sample of psychiatric partici-pants included 164 men (age range = 26-85 years, M = 55.44

Forbey et al. 451

years, SD = 11.90) who were Caucasian (n = 138, 84.7%), African American (n = 21, 12.9%), or of another or uniden-tified ethnicity (n = 5, 3.1%).

MeasuresMinnesota Multiphasic Personality Inventory–2–Restructured

Form. Revised from the MMPI-2 (Butcher et al., 2001), the MMPI-2-RF (Ben-Porath & Tellegen, 2008; Tellegen & Ben-Porath, 2008/2011) is a 338-item, true/false, self-report inventory that assesses an individual’s psychological func-tioning in a number of domains (i.e., personality, psychopa-thology, and social/ behavioral functioning). The MMPI-2-RF Technical Manual (Tellegen & Ben-Porath, 2008/2011) pro-vides extensive evidence supporting reliability and validity for this instrument. For the current study, the MMPI-2-RF was rescored from a conventional administration of the MMPI-2 (see Procedure section below). Tellegen and Ben-Porath (2008/2011) and Van der Heijden, Egger, and Derksen (2010) have demonstrated that MMPI-2-RF scale scores gen-erated from an MMPI-2 administration are interchangeable with those generated from the MMPI-2-RF booklet.

Collateral measures. Eight self-report based collateral mea-sures, originally selected for the purpose of examining the comparative validity of modularized versions of the MMPI-2-CA and conventional administrations of the MMPI-2 with conceptually relevant personality and psychopathology con-structs were used in the current study. Depending on setting, the collateral measures were either divided into two adminis-tration packets (i.e., the college and psychiatric participants) or all measures were administered in a single packet (i.e., the correctional participants). All eight collateral measures were administered via paper and pencil. Each of the eight self-report collateral measures is described below.

Barratt Impulsivity Scale–Version 10. The Barratt Impulsiv-ity Scale (BIS, Barratt, 1985) is a 34-item measure of impulsiv-ity rated on a 4-point Likert-type scale (1 = rarely/never, 4 = almost always/always) that provides a total score as well as scores on three dimensions of impulsivity: Nonplanning (12 items), Motor (11 items), and Cognitive (11 items). In the current study, only the total scale score was uses, which had an estimated internal consistency (α) of .80 for college, .87 for correctional, and .86 for psychiatric participants.

Beck Depression Inventory. The Beck Depression Inven-tory (BDI; Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) is a 21-item measure of depressive symptomatology rated on a 4-point Likert-type scale, with higher ratings generally indicating higher levels of psychological distress. In the current study, estimated internal consistency (α) for the BDI was .88 for the college, .89 for the correctional, and .93 for the psychiatric participants.

Drug Abuse Screening Test. The Drug Abuse Screening Test (DAST; Skinner, 1982) is a 20-item measure rated dichotomously (i.e., yes or no) that assesses an individual’s

self-reported consumption of prescription, over the counter, and other illicit drugs. In the current study, estimated inter-nal consistency (α) for the DAST was estimated to be .81 for the college, .93 for the correctional, and .94 for the psy-chiatric participants.

Magical Ideation Scale. The Magical Ideation Scale (MIS; Eckblad & Chapman, 1983) is a 30-item true/false inven-tory that assesses beliefs about causal relations between events that are unconventional and which are commonly associated with thought disorders. In the current study, esti-mated internal consistency (α) for the MIS was .80 for the college, .81 for the correctional, and .86 for the psychiatric participants.

Michigan Alcohol Screening Test. The Michigan Alcohol Screening Test (MAST; Selzer, 1971) is a 24-item measure rated dichotomously (i.e., yes or no) that assesses an indi-vidual’s self-reported level of problematic alcohol con-sumption. In the current study, internal consistency (α) for the MAST was estimated to be .68 for the college, .80 for the correctional, and .84 for the psychiatric participants.

Perceptual Aberration Scale. The Perceptual Aberration Scale (PAS; Chapman, Chapman, & Raulin, 1978) is a 35-item true/false inventory that assesses physical and other perceptual distortions related to thought disorders. In the current study, internal consistency (α) for the PAS was esti-mated to be .96 for the college, .97 for the correctional, and .92 for the psychiatric participants.

Screener for Somatoform Disorders. The Screener for Somatoform Disorders (SSD; Janca et al., 1995) is a 12-item measure designed to assess diffuse somatic complaints related to somatoform disorders as defined by the Interna-tional Classification of Diseases (ICD-10; World Health Organization, 1992) and the fourth edition of the American Psychiatric Association’s (APA) Diagnostic and Statistical Manual of Mental Disorders (DSM-IV; APA, 1994). In the current study, internal consistency (α) for the SSD was esti-mated to be .80 for the college, .84 for the correctional, and .86 for the psychiatric participants.

State-Trait Personality Inventory. The State-Trait Personal-ity Inventory (STPI; Spielberger, 1979) is a 60-item mea-sure of anxiety and anger rated on a 4-point Likert-type scale (1 = almost never, 4 = almost always). The current study relied on a modified 10-item version of the instrument in which only the Trait Anger items were administered. In the current study, the estimated internal consistency (α) for Trait Anger subscale was .69 for the college, .88 for the cor-rectional, and .88 for the psychiatric participants.

ProcedureAs indicated, all participants in the current study were selected from three larger unpublished studies examining various modularized versions of the computerized adaptive version of the MMPI-2 (i.e., the MMPI-2-CA). Although

452 Assessment 20(4)

the measures used in each study were consistent across set-tings, because of various institutional considerations/con-straints, each of the data collection sites had slightly different data collection procedures. However, all partici-pants were assessed in accordance with procedures approved by the institutional review board at the facilities where the data were collected, received compensation in line with those procedures dictated by the facilities and their respective institutional review boards, and were free to withdraw their participation at any time.

For the college participants, after agreeing to participate, individuals were assigned to complete the MMPI-2, as well as one of two packets of selected collateral measures, and then return in 1 week to complete another MMPI-2 admin-istration and the remaining collateral measures. For the psy-chiatric participants, letters were sent prior to initial clinic appointments introducing the study and individuals were asked to call and set up an appointment to participate in the study. However, the actual data collection in the psychiatric participants was similar to that of the college participants, with the exception that the time frame for the second MMPI-2 administration ranged from 6 to 10 days. For both college and psychiatric participants, administration of the MMPI-2 and collateral measures was counterbalanced across the two time frames (i.e., either the adaptive or con-ventional version of the MMPI-2 was administered fol-lowed by the collateral measures or vice versa).

For the correctional participants, all individuals com-pleted a standard paper and pencil audio-taped version of the MMPI-2 on intake to the correctional facility as part of stan-dard institutional screening procedures. On completion of the paper-and-pencil audio MMPI-2, participants were recruited for participation in the larger study. Those who volunteered for the study were required to have a sixth-grade reading level, which was verified using facility records.2 Administration of the collateral measures (as well as either a computerized conventional or adaptive MMPI-2) occurred in a group format typically between 1 and 5 days after administration of the intake MMPI-2, with 87.8% (N = 555) of participants completing the collateral measures 1 day after completing the initial MMPI-2. The administration order of the second MMPI-2 and collateral measures was counterbalanced.

In the current study, across the samples, only the first conventional MMPI-2 administered was used in the rescor-ing of the MMPI-2 to the MMPI-2-RF. In the case of the correctional participants, the rescored MMPI-2 selected for use in the current study was always the first MMPI-2 administered during the original intake procedure. For the college and psychiatric participants, the rescored MMPI-2 used in the current study could have been administered either approximately a week before or after the administra-tion of some of the collateral measures because of counter-balancing of administration formats. Across settings, all

collateral measures were administered via a Latin square design and were considered invalid for the current study’s analyses if 10% or more of items were not answered.

Response Style Group AssignmentAfter removing content nonresponsive MMPI-2-RF profiles (as described in the Participants section above), participants in each sample were assigned to one of three groups: CBIR-OR, CBIR-UR, and WNL reporting on the validity scales, based on their T scores on standard MMPI-2-RF validity. The MMPI-2-RF Validity scale T score cutoffs used for group assignment in the current study were consistent with those indicated in the MMPI-2-RF Manual for Administration, Scoring, and Interpretation (Ben-Porath & Tellegen, 2008). CBIR-OR groups were identified by T scores of 100 or greater on the Infrequent Responses (F-r), Infrequent Somatic Responses (F

S), Symptom Validity (FBS-

r), or Response Bias (RBS) Scales, and/or 80 or more on the Infrequent Psychopathology Responses (F

P-r) Scale.

CBIR-UR was identified by T scores of 70 or greater on the Uncommon Virtues (L-r) scale and/or an Adjustment Validity (K-r) scale score of 66 or higher. In addition, no individuals in the CBIR-OR group could have elevations on the under-reporting scales and vice versa. Finally, the WNL group consisted of individuals who had no elevations on any of the over-reporting and/or under-reporting MMPI-2-RF scales.

Using this group assignment procedure for the 1,065 included college participants, 100 were assigned to CBIR-OR, 59 were assigned to CBIR-UR, 903 were assigned to WNL, and 3 were removed from subsequent analyses because of having elevations on both CBIR-OR and CBIR-UR Scales. For the 613 included correctional participants, 39 were assigned to CBIR-OR, 163 were assigned to CBIR-UR, 406 were assigned to WNL, and 5 were removed from subsequent analyses because of having elevations on both CBIR-OR and CBIR-UR Scales. For the 164 included psychiatric participants, 44 were assigned to CBIR-OR, 16 were assigned to CBIR-UR, and 104 were assigned to WNL.

Comparative demographic analyses in group assign-ments revealed a few statistically significant differences for each of the samples. For college participants, no differences emerged in terms of gender or ethnic/racial identity. However, statistically significant differences of a small effect size (Cohen, 1988) was demonstrated for age, F(2, 1059) = 3.409, p < .033, η2 = .01. Post hoc comparisons for age using Tukey’s HSD test indicated that the mean age in the CBIR-UR group (M = 20.61 years, SD = 5.275) was significantly older than the WNL (M = 19.57 years, SD = 3.052) and CBIR-OR groups (M = 19.35 years, SD = 2.037) groups. For both correctional and psychiatric participants, no differences were found between groups in terms of age or ethnicity.

Forbey et al. 453

Data Analyses

To examine potential differences in MMPI-2-RF scale scores between the three response style groups that were created, a series of t tests were computed comparing mean T scores on MMPI-2-RF substantive scales for each of the three samples of participants separately. To reduce potential Type I error, a Bonferroni correction was applied and the critical alpha for analyses within each of these three sam-ples was set to .001 (.05/42). Cohen’s d (1988) effect sizes, with .3, .5, and .8 reflecting small, medium, and large effects, respectively, for the differences between means were reported for all analyses, regardless of significance.

To examine for potential differences on collateral mea-sure scores between response style groups, a series of t tests were computed comparing mean scores on the collateral measures for each of the three samples. As each of the eight collateral measures have unique scale score ranges, means, and standard deviations, the descriptive statistics reported for the collateral measure analyses were converted (within each group) to a z-score metric to facilitate interpretation of the results. As with the MMPI-2-RF substantive scale anal-yses, to reduce potential Type I error, a Bonferroni correc-tion was applied, setting the critical alpha for these analyses at .006 (.05/8) for each of the samples. Effect sizes (d; Cohen, 1988) were reported for all analyses, regardless of significance.

ResultsThe first set of analyses explored the differences on sub-stantive scale scores between the WNL and CBIR-OR response style group as well as for the WNL and CBIR-UR response style groups, across the three samples. Tables 1, 2, and 3 contain the results of these analyses for the college, correctional, and psychiatric participants, respectively.

After a Bonferroni correction, t-test results for the 42 comparisons of MMPI-2-RF substantive scale scores for the CBIR-OR and WNL groups indicated that 36 scale scores for college participants, 36 scale scores for correc-tional participants, and 35 scale scores for psychiatric par-ticipants were significantly different. In all cases, mean MMPI-2-RF substantive scale scores for the CBIR-OR groups were significantly higher than the WNL groups. Median effect sizes for the CBIR-OR and WNL mean scale score comparisons for the Higher Order (H-O) scales were 1.05 for college participants (range = 0.91 to 1.57), 2.08 for correctional participants (range = 0.63 to 2.46), and 1.56 for the psychiatric participants (range = 0.75 to 1.86). Median effect sizes for the CBIR-OR and WNL mean scale score comparisons for the Restructured Clinical scales were 1.07 for college participants (range = 0.71 to 1.49), 2.07 for cor-rectional participants (range = 0.72 to 2.34), and 1.37 for the psychiatric participants (range = 0.82 to 2.01). For the

Specific Problems and Interest scales, median effect sizes for the CBIR-OR and WNL scale score comparisons were .70 for college participants (range = 0.10 to 1.45), 1.41 for correctional participants (range = −0.31 to 2.99), and 1.07 for the psychiatric participants (range = −0.05 to 1.87). Finally, median effect sizes for the CBIR-OR and WNL comparisons for scores on the PSY-5-r scales were 0.59 for college participants (range = 0.24 to 1.64), 1.26 for correc-tional participants (range = −0.42 to 2.49), and 0.55 for the psychiatric participants (range = 0.24 to 1.85). For the CBIR-OR and WNL effect size analyses, nonsignificant results were included in median calculations.

For the comparison of mean MMPI-2-RF substantive scale scores between CBIR-UR and WNL response style groups for each of the three samples, after a Bonferroni correction, t-test results for the 42 comparisons indicated that 30 scale scores for college participants, 32 scale scores for correctional participants, and 16 scale scores for psy-chiatric participants were significantly different. When sig-nificant differences existed, mean scores for the CBIR-UR groups were significantly lower than the WNL responders’ mean scores. Median effect sizes for the CBIR-UR and WNL mean scale score comparisons for the H-O scales were −0.65 for college participants (range = −0.50 to −1.04), −0.96 for correctional participants (range = −.49 to −1.09), and −0.74 for the psychiatric participants (range = −0.57 to −1.63). Median effect sizes for the CBIR-UR and WNL mean scale score comparisons for the Restructured Clinical scales were −0.61 for college participants (range = −0.28 to −1.13), −0.73 for correctional participants (range = −0.39 to −1.05), and −0.92 for the psychiatric participants (range = −0.37 to −1.42). For the Specific Problems and Interest scales, median effect sizes for the CBIR-UR and WNL scale score comparisons were −0.51 for college par-ticipants (range = 0.20 to −0.97), −0.54 for correctional participants (range = 0.24 to −0.85), and −0.73 for the psy-chiatric participants (range = 0.35 to −1.23). Finally, median effect sizes for the CBIR-UR and WNL compari-sons for scores on the PSY-5-r scales were −0.43 for col-lege participants (range = 0.14 to −1.13), −0.50 for correctional participants (range = −0.07 to −0.97), and −0.45 for the psychiatric participants (range = 0.04 to −1.39). As with the CBIR-OR analyses, for the CBIR-UR and WNL effect size analyses nonsignificant results were included in median calculations.

Tables 4, 5, and 6 report the scores on the collateral mea-sures for the CBIR-OR and WNL response style groups, as well as CBIR-UR and WNL response style groups for the college, correctional, and psychiatric participants, respec-tively. Within the tables, total scores for collateral measures are reported in z scores, and the measures are divided into three broad categorizations (i.e., externalizing, internaliz-ing, and thought dysfunction) based on the general content and/or the construct examined by each collateral measure.

454 Assessment 20(4)

Table 1. Minnesota Multiphasic Personality Inventory–2 (MMPI-2) Scale Mean T Score Differences Between MMPI-2 Within Normal Limits (WNL) Versus Content-Responsive Invalidity: Over-Reporting (CBIR-OR) and Content Responsive Invalidity: Under-Reporting (CBIR-UR): College Participants.

MMPI scales

WNL (n = 903)

CBIR-OR, (n = 100)

t df p ES (d)

CBIR-UR, (n = 59)

t df p ES (d)Mean SD Mean SD Mean SD

Higher order EID 51.04 10.74 62.50 12.34 −9.97 1001 <.001 1.05 40.08 7.11 7.72 960 <.001 −1.04 THD 52.52 9.92 68.85 14.17 −14.87 1001 <.001 1.57 47.58 9.46 3.72 960 <.001 −0.50 BXD 54.09 9.79 63.04 10.06 −8.65 1001 <.001 0.91 47.80 8.95 4.81 960 <.001 −0.65Restructured clinical RCd 54.54 10.07 65.51 11.60 −10.17 1001 <.001 1.07 43.44 6.42 8.36 960 <.001 −1.12 RC1 53.17 10.29 64.56 11.70 −10.35 1001 <.001 1.09 47.17 7.55 4.40 960 <.001 −0.59 RC2 48.03 9.85 55.32 13.15 −6.77 1001 <.001 0.71 45.25 8.32 2.11 960 ns −0.28 RC3 56.14 9.40 63.80 9.40 −7.73 1001 <.001 0.81 46.66 9.08 7.52 960 <.001 −1.01 RC4 52.87 9.63 62.00 10.59 −8.90 1001 <.001 0.94 47.07 8.75 4.51 960 <.001 −0.61 RC6 55.39 10.64 70.12 13.42 −12.77 1001 <.001 1.35 49.04 7.71 4.52 960 <.001 −0.61 RC7 53.94 10.88 68.58 12.83 −12.53 1001 <.001 1.32 41.83 7.00 8.44 960 <.001 −1.13 RC8 54.10 10.47 70.08 13.05 −14.10 1001 <.001 1.49 48.05 10.60 4.29 960 <.001 −0.58 RC9 57.09 10.90 67.21 10.95 −8.80 1001 <.001 0.93 46.59 8.62 7.25 960 <.001 −0.97Somatic/cognitive MLS 51.38 9.62 59.24 11.38 −7.61 1001 <.001 0.80 44.64 6.17 5.31 960 <.001 −0.71 GIC 52.55 11.65 57.98 15.19 −4.28 1001 <.001 0.45 48.10 6.59 2.90 960 ns −0.39 HPC 52.20 10.65 59.64 11.56 −6.57 1001 <.001 0.69 46.56 7.15 4.01 960 <.001 −0.54 NUC 54.14 10.84 67.04 11.82 −11.18 1001 <.001 1.18 48.73 10.07 3.73 960 <.001 −0.50 COG 55.92 11.63 71.06 13.39 −12.15 1001 <.001 1.28 46.93 8.79 5.83 960 <.001 −0.78Internalizing SUI 49.32 10.72 60.85 17.78 −9.43 1001 <.001 0.99 45.58 4.43 2.67 960 ns −0.36 HLP 50.22 10.66 59.49 12.53 −8.09 1001 <.001 0.85 44.88 6.60 3.80 960 <.001 −0.51 SFD 53.01 11.65 60.75 13.00 −6.23 1001 <.001 0.66 44.58 5.95 5.51 960 <.001 −0.74 NFC 54.03 10.61 61.50 10.83 −6.66 1001 <.001 0.70 43.95 6.79 7.20 960 <.001 −0.97 STW 52.88 10.89 61.08 11.86 −7.08 1001 <.001 0.75 43.53 5.40 6.54 960 <.001 −0.88 AXY 55.60 12.70 73.63 14.24 −13.30 1001 <.001 1.40 47.34 7.59 4.94 960 <.001 −0.66 ANP 53.24 10.74 63.24 10.85 −8.83 1001 <.001 0.93 43.97 7.58 6.52 960 <.001 −0.88 BRF 52.70 10.93 69.14 14.38 −13.78 1001 <.001 1.45 47.42 8.21 3.64 960 <.001 −0.49 MSF 48.40 8.60 51.54 10.22 −3.39 1001 <.001 0.36 44.75 7.07 3.20 960 <.001 −0.43Externalizing JCP 50.61 10.12 57.10 11.11 −6.02 1001 <.001 0.63 46.97 9.07 2.69 960 ns −0.36 SUB 52.66 11.29 64.16 13.89 −9.42 1001 <.001 0.99 49.22 9.39 2.29 960 ns −0.31 AGG 52.27 11.60 62.13 12.31 −8.01 1001 <.001 0.84 43.29 7.08 5.88 960 <.001 −0.79 ACT 53.74 9.96 60.54 10.32 −6.46 1001 <.001 0.68 46.76 8.94 5.24 960 <.001 −0.70Interpersonal FML 51.56 10.18 59.77 11.35 −7.56 1001 <.001 0.80 44.00 6.94 5.62 960 <.001 −0.75 IPP 45.48 8.11 45.69 6.57 −0.25 1001 ns 0.03 47.08 6.56 −1.48 960 ns 0.20 SAV 45.46 9.87 47.11 11.10 −1.56 1001 ns 0.17 44.93 9.19 0.40 960 ns −0.05 SHY 50.83 11.10 55.18 10.55 −4.35 1001 <.001 0.39 43.12 6.16 5.28 960 <.001 −0.71 DSF 50.87 9.73 57.18 12.67 −6.31 1001 <.001 0.63 50.17 8.84 0.54 960 ns −0.07Interest AES 44.80 9.48 45.75 10.28 −0.95 1001 ns 0.10 45.27 9.53 −0.37 960 ns 0.05 MEC 49.10 9.82 50.37 9.41 −1.27 1001 ns 0.13 48.86 10.34 0.18 960 ns −0.02Personality Psychopathology Five (PSY-5) AGGR-r 51.43 10.18 54.39 9.61 −2.96 1001 ns 0.29 47.81 6.92 2.69 960 ns −0.36 PSYC-r 52.90 10.02 69.95 13.10 −17.05 1001 <.001 1.64 47.88 9.66 3.74 960 <.001 −0.50 DISC-r 54.30 10.00 60.15 9.63 −5.85 1001 <.001 0.59 49.98 10.09 3.21 960 <.001 −0.43 NEGE-r 54.16 11.19 66.43 13.76 −12.27 1001 <.001 1.07 41.75 6.83 8.42 960 <.001 −1.13 INTR-r 44.28 9.56 46.66 11.07 −2.37 1001 ns 0.24 45.58 9.98 −1.01 960 ns 0.14

Note. ES = effect size; Higher Order: EID = Emotional/Internalizing Dysfunction, THD = Thought Dysfunction, BXD = Behavioral/Externalizing Dysfunction; Restructured Clini-cal (RC): RCd = Demoralization, RC1 = Somatic Complaints, RC2 = Low Positive Emotions, RC3 = Cynicism, RC4 = Antisocial Behavior, RC6 = Ideas of Persecution, RC7 = Dysfunctional Negative Emotions, RC8 = Aberrant Experiences, RC9 = Hypomanic Activation; Somatic/Cognitive: MLS = Malaise, GIC = Gastrointestinal Complaints, HPC = Head Pain Complaints, NUC = Neurological Complaints, COG = Cognitive Complaints; Internalizing Scales: SUI = Suicidal/Death Ideation, HLP = Helplessness/Hopelessness, SFD = Self-Doubt, NFC = Inefficacy, STW = Stress/Worry, AXY = Anxiety, ANP = Anger Proneness, BRF = Behavior-Restricting Fears, MSF = Multiple Specific Fears; External-izing: JCP = Juvenile Conduct Problems, SUB = Substance Abuse, AGG = Aggression, ACT = Activation; Interpersonal Scales: FML = Family Problems, IPP = Interpersonal Passivity, SAV = Social Avoidance, SHY = Shyness, DSF = Disaffiliativeness; Interest Scales: AES = Aesthetic-Literary Interests, MEC = Mechanical-Physical Interests; Personal-ity Psychopathology Five (PSY-5): AGGR-r = Aggressiveness-Revised, PSYC-r = Psychoticism–Revised, DISC-r = Disconstraint–Revised, NEGE-r = Negative Emotionality/Neuroticism–Revised, INTR-r = Introversion/Low Positive Emotionality–Revised.

Forbey et al. 455

Table 2. Minnesota Multiphasic Personality Inventory–2 (MMPI-2) Scale Mean T Score Differences Between MMPI-2 Within Normal Limits (WNL) Versus Content-Responsive Invalidity: Over-Reporting (CBIR-OR) and Content Responsive Invalidity: Under-Reporting (CBIR-UR): Correctional Participants.

MMPI scales

WNL (n = 406)

CBIR-OR (n = 39)

t df p ES (d)

CBIR-UR (n = 163)

t df p ES (d)Mean SD Mean SD Mean SD

Higher order EID 47.80 10.46 70.31 14.08 −12.41 443 <.001 2.08 38.75 6.06 10.34 567 <.001 −0.96 THD 51.17 9.34 75.51 14.53 −14.67 443 <.001 2.46 46.63 8.82 5.33 567 <.001 −0.49 BXD 64.47 10.92 71.38 11.16 −3.77 443 <.001 0.63 52.90 9.82 11.76 567 <.001 −1.09Restructured clinical RCd 51.10 9.64 72.00 11.86 −12.66 443 <.001 2.12 42.55 5.74 10.56 567 <.001 −0.98 RC1 48.05 10.14 70.10 14.29 −12.50 443 <.001 2.09 44.34 8.18 4.15 567 <.001 −0.39 RC2 47.32 10.01 68.56 16.02 −11.89 443 <.001 1.99 41.98 6.66 6.27 567 <.001 −0.58 RC3 56.99 11.18 65.21 10.87 −4.40 443 <.001 0.74 48.82 11.08 8.00 567 <.001 −0.73 RC4 66.69 11.33 76.87 9.90 −5.41 443 <.001 0.91 54.92 10.85 11.34 567 <.001 −1.05 RC6 56.44 11.74 81.82 12.78 −12.80 443 <.001 2.14 50.25 10.10 5.91 567 <.001 −0.55 RC7 46.84 9.24 66.74 12.86 −12.36 443 <.001 2.07 39.20 5.78 9.81 567 <.001 −0.91 RC8 49.81 8.33 71.15 15.16 −13.96 443 <.001 2.34 44.70 7.73 6.75 567 <.001 −0.63 RC9 50.82 9.44 57.82 12.05 −4.31 443 <.001 0.72 42.68 6.71 10.03 567 <.001 −0.93Somatic/cognitive MLS 50.93 9.36 48.78 7.96 −9.19 443 <.001 −0.23 45.44 6.91 6.79 567 <.001 −0.63 GIC 48.78 7.96 61.28 15.71 −8.38 443 <.001 1.41 47.36 5.24 2.10 567 ns −0.19 HPC 48.48 9.21 65.26 11.64 −10.60 443 <.001 1.78 45.48 6.66 3.78 567 <.001 −0.35 NUC 50.45 10.69 70.21 16.97 −10.37 443 <.001 1.74 46.00 7.51 4.86 567 <.001 −0.45 COG 49.88 11.24 84.41 14.45 −13.19 443 <.001 2.99 42.81 5.94 7.62 567 <.001 −0.71Internalizing SUI 46.87 6.42 65.46 17.88 −13.74 443 <.001 2.30 45.39 2.83 2.83 567 ns −0.26 HLP 46.07 9.27 62.10 13.32 −9.88 443 <.001 1.66 42.01 4.71 5.32 567 <.001 −0.49 SFD 48.97 9.14 64.64 12.43 −10.50 443 <.001 1.66 42.79 2.82 8.48 567 <.001 −0.79 NFC 48.27 8.80 64.05 10.99 −10.45 443 <.001 1.75 41.88 5.93 8.52 567 <.001 −0.79 STW 49.50 9.72 63.31 12.39 −8.26 443 <.001 1.38 41.80 7.09 9.17 567 <.001 −0.85 AXY 50.90 11.09 72.82 13.85 −11.52 443 <.001 1.93 45.66 4.72 5.82 567 <.001 −0.54 ANP 49.20 10.61 60.64 11.67 −6.37 443 <.001 1.07 42.49 5.92 7.61 567 <.001 −0.71 BRF 46.67 7.06 57.64 12.14 −8.59 443 <.001 1.44 45.42 5.87 1.99 567 ns −0.19 MSF 45.92 7.87 46.08 8.51 −0.12 443 ns 0.02 45.48 6.80 0.63 567 ns −0.06Externalizing JCP 67.18 12.31 74.72 8.71 −3.73 443 <.001 0.63 57.00 12.21 8.94 567 <.001 −0.83 SUB 62.71 13.90 69.10 13.55 −2.73 443 ns 0.46 53.38 13.09 7.36 567 <.001 −0.68 AGG 51.24 11.84 60.97 11.47 −4.92 443 <.001 0.82 41.70 7.92 9.47 567 <.001 −0.88 ACT 47.03 9.78 56.28 12.88 −5.47 443 <.001 0.92 41.75 7.60 6.19 567 <.001 −0.57Interpersonal FML 48.69 10.30 65.67 14.33 −9.46 443 <.001 1.59 42.28 6.53 7.38 567 <.001 −0.68 IPP 43.84 7.09 52.64 14.49 −6.57 443 <.001 1.10 45.45 5.90 −2.58 567 ns 0.24 SAV 48.05 10.15 58.64 15.42 −5.90 443 <.001 0.99 47.79 8.41 0.29 567 ns −0.03 SHY 45.99 9.02 59.64 13.20 −8.62 443 <.001 1.44 41.24 6.41 6.13 567 <.001 −0.57 DSF 48.15 8.83 64.72 17.99 −9.93 443 <.001 1.66 46.74 7.40 1.81 567 ns −0.17Interest AES 44.52 8.05 47.05 10.44 −1.82 443 ns 0.31 45.35 9.83 −1.04 567 ns 0.10 MEC 63.85 9.05 61.00 9.69 1.87 443 ns −0.31 62.54 9.53 1.54 567 ns −0.14Personality Psychopathology Five (PSY-5) AGGR-r 56.56 9.23 52.44 15.08 2.35 443 ns −0.42 52.25 7.49 5.00 567 <.001 −0.49 PSYC-r 49.62 9.40 74.33 14.37 −14.85 443 <.001 2.49 45.05 8.61 5.63 567 <.001 −0.50 DISC-r 65.54 10.71 71.28 11.09 −3.19 443 ns 0.53 55.34 10.17 10.43 567 <.001 −0.97 NEGE-r 49.47 9.90 66.05 12.19 −9.78 443 <.001 1.64 40.68 7.05 10.33 567 <.001 −0.96 INTR-r 48.12 9.83 61.31 16.03 −7.49 443 <.001 1.26 47.48 7.36 0.75 567 ns −0.07

Note. ES = effect size; Higher Order: EID = Emotional/Internalizing Dysfunction, THD = Thought Dysfunction, BXD = Behavioral/Externalizing Dysfunction; Restructured Clini-cal (RC): RCd = Demoralization, RC1 = Somatic Complaints, RC2 = Low Positive Emotions, RC3 = Cynicism, RC4 = Antisocial Behavior, RC6 = Ideas of Persecution, RC7 = Dysfunctional Negative Emotions, RC8 = Aberrant Experiences, RC9 = Hypomanic Activation; Somatic/Cognitive: MLS = Malaise, GIC = Gastrointestinal Complaints, HPC = Head Pain Complaints, NUC = Neurological Complaints, COG = Cognitive Complaints; Internalizing Scales: SUI = Suicidal/Death Ideation, HLP = Helplessness/Hopelessness, SFD = Self-Doubt, NFC = Inefficacy, STW = Stress/Worry, AXY = Anxiety, ANP = Anger Proneness, BRF = Behavior-Restricting Fears, MSF = Multiple Specific Fears; External-izing: JCP = Juvenile Conduct Problems, SUB = Substance Abuse, AGG = Aggression, ACT = Activation; Interpersonal Scales: FML = Family Problems, IPP = Interpersonal Passivity, SAV = Social Avoidance, SHY = Shyness, DSF = Disaffiliativeness; Interest Scales: AES = Aesthetic-Literary Interests, MEC = Mechanical-Physical Interests; Personal-ity Psychopathology Five (PSY-5): AGGR-r = Aggressiveness–Revised, PSYC-r = Psychoticism–Revised, DISC-r = Disconstraint–Revised, NEGE-r = Negative Emotionality/Neuroticism–Revised, INTR-r = Introversion/Low Positive Emotionality–Revised.

456 Assessment 20(4)

Table 3. Minnesota Multiphasic Personality Inventory–2 (MMPI-2) Scale Mean T Score Differences between MMPI-2 Within Normal Limits (WNL) Versus Content-Responsive Invalidity: Over-Reporting (CBIR-OR) and Content-Responsive Invalidity: Under-Reporting (CBIR-UR): Psychiatric Participants.

MMPI scales

WNL (n = 104)

CBIR-OR (n = 44)

t df p ES (d)

CBIR-UR (n = 16)

t df p ES (d)Mean SD Mean SD Mean SD

Higher order EID 58.21 12.12 76.45 10.73 −8.65 146 <.001 1.56 39.44 6.11 6.06 118 <.001 −1.63 THD 52.67 10.53 77.20 18.01 −10.35 146 <.001 1.86 46.88 7.14 2.13 118 ns −0.57 BXD 56.54 11.36 64.95 10.95 −4.16 146 <.001 0.75 48.38 8.75 2.75 118 ns −0.74Restructured clinical RCd 59.04 11.93 75.84 9.81 −8.42 146 <.001 1.48 43.00 5.44 5.28 118 <.001 −1.42 RC1 59.97 12.38 76.93 12.37 −7.62 146 <.001 1.37 50.38 9.05 2.98 118 ns −0.80 RC2 59.68 13.70 73.25 14.95 −5.34 146 <.001 0.96 47.44 6.37 3.51 118 <.001 −0.94 RC3 53.93 11.66 66.14 12.19 −5.74 146 <.001 1.03 41.81 8.72 3.98 118 <.001 −1.07 RC4 58.86 11.15 68.41 12.56 −4.59 146 <.001 0.82 48.44 12.72 3.42 118 <.001 −0.92 RC6 54.54 11.32 78.34 15.71 −10.37 146 <.001 1.86 50.44 7.88 1.40 118 ns −0.37 RC7 52.36 10.53 73.70 10.90 −4.93 146 <.001 2.01 38.94 5.42 4.98 118 <.001 −1.34 RC8 54.22 11.49 77.36 13.72 −4.57 146 <.001 1.90 46.50 7.27 2.60 118 ns −0.70 RC9 50.49 10.26 59.95 11.59 −4.17 146 <.001 0.89 42.13 5.94 3.17 118 ns −0.85Somatic/cognitive MLS 65.52 12.41 75.48 11.43 −4.57 146 <.001 0.82 52.94 14.86 3.68 118 <.001 −0.99 GIC 54.50 13.37 64.82 14.66 −4.17 146 <.001 0.75 49.38 7.26 1.50 118 ns −0.40 HPC 55.11 11.33 69.93 12.44 −7.07 146 <.001 1.27 49.69 7.69 1.85 118 ns −0.50 NUC 63.78 14.70 79.18 12.82 −6.04 146 <.001 1.09 53.50 11.11 2.68 118 ns −0.72 COG 59.99 14.38 82.18 9.67 −9.37 146 <.001 1.69 44.25 7.22 4.28 118 <.001 −1.15Internalizing SUI 57.97 13.34 78.05 19.32 −8.03 146 <.001 1.31 46.31 5.25 2.38 118 ns −0.93 HLP 57.02 13.14 72.12 11.43 −6.63 146 <.001 1.19 44.75 7.62 3.63 118 <.001 −0.98 SFD 56.21 11.74 68.05 9.28 −5.94 146 <.001 1.07 42.63 2.50 4.60 118 <.001 −1.23 NFC 54.65 11.88 67.91 8.70 −6.68 146 <.001 1.20 42.56 4.93 4.01 118 <.001 −1.08 STW 51.45 11.16 66.68 9.89 −7.84 146 <.001 1.41 40.06 4.95 4.01 118 <.001 −1.08 AXY 54.67 12.88 80.41 15.74 −10.38 146 <.001 1.87 45.88 5.12 2.69 118 ns −0.72 ANP 52.06 11.10 65.52 10.03 −6.93 146 <.001 1.25 40.50 3.22 4.12 118 <.001 −1.11 BRF 48.67 8.23 63.64 11.88 −8.81 146 <.001 1.58 46.69 6.78 0.918 118 ns −0.25 MSF 47.68 8.44 49.95 9.17 −1.46 146 <.001 0.26 43.00 10.51 2.00 118 ns −0.54Externalizing JCP 57.96 12.74 65.84 12.76 −3.44 146 <.001 0.62 51.13 14.82 1.96 118 ns −0.52 SUB 55.27 10.91 61.39 15.24 −2.76 146 ns 0.50 50.50 11.64 1.61 118 ns −0.43 AGG 50.95 11.74 64.36 14.23 −5.96 146 <.001 1.07 40.88 4.76 3.38 118 <.001 −0.91 ACT 48.92 9.15 55.32 10.75 −3.69 146 <.001 0.66 41.75 9.33 2.91 118 ns −0.78Interpersonal FML 51.61 10.85 68.52 10.80 −8.68 146 <.001 1.56 41.56 5.78 3.62 118 <.001 −0.97 IPP 48.17 9.37 50.05 10.51 −1.07 146 ns 0.19 45.56 4.93 1.09 118 ns −0.29 SAV 55.03 11.92 63.18 14.03 −3.60 146 <.001 0.65 46.81 4.74 2.72 118 ns −0.73 SHY 51.64 11.38 58.95 11.44 −3.57 146 <.001 0.64 43.00 6.40 2.96 118 ns −0.79 DSF 51.72 11.54 67.68 17.67 −6.51 146 <.001 1.17 45.75 4.78 2.03 118 ns −0.55Interest AES 46.87 10.06 46.34 9.03 0.30 146 ns −0.05 50.31 8.51 −1.30 118 ns 0.35 MEC 56.90 9.75 56.61 9.11 0.17 146 ns −0.03 60.06 8.66 −1.22 118 ns 0.33Personality Psychopathology Five (PSY-5) AGGR-r 49.48 9.49 51.84 11.03 −1.32 146 ns 0.24 49.88 9.49 −0.16 118 ns 0.04 PSYC-r 52.20 10.62 76.52 18.55 −10.05 146 <.001 1.81 47.56 6.83 1.69 118 ns −0.45 DISC-r 56.35 10.44 61.98 10.67 −2.98 146 <.001 0.54 51.81 9.11 1.64 118 ns −0.44 NEGE-r 53.90 10.78 73.36 9.77 −10.31 146 ns 1.85 39.50 6.36 5.20 118 <.001 −1.39 INTR-r 56.78 13.82 64.77 15.91 −3.07 146 ns 0.55 48.50 5.88 2.36 118 ns −0.63

Note. ES = effect size; Higher Order: EID = Emotional/Internalizing Dysfunction, THD = Thought Dysfunction, BXD = Behavioral/Externalizing Dysfunction; Restructured Clini-cal (RC): RCd = Demoralization, RC1 = Somatic Complaints, RC2 = Low Positive Emotions, RC3 = Cynicism, RC4 = Antisocial Behavior, RC6 = Ideas of Persecution, RC7 = Dysfunctional Negative Emotions, RC8 = Aberrant Experiences, RC9 = Hypomanic Activation; Somatic/Cognitive: MLS = Malaise, GIC = Gastrointestinal Complaints, HPC = Head Pain Complaints, NUC = Neurological Complaints, COG = Cognitive Complaints; Internalizing Scales: SUI = Suicidal/Death Ideation, HLP = Helplessness/Hopelessness, SFD = Self-Doubt, NFC = Inefficacy, STW = Stress/Worry, AXY = Anxiety, ANP = Anger Proneness, BRF = Behavior-Restricting Fears, MSF = Multiple Specific Fears; External-izing: JCP = Juvenile Conduct Problems, SUB = Substance Abuse, AGG = Aggression, ACT = Activation; Interpersonal Scales: FML = Family Problems, IPP = Interpersonal Passivity, SAV = Social Avoidance, SHY = Shyness, DSF = Disaffiliativeness; Interest Scales: AES = Aesthetic-Literary Interests, MEC = Mechanical–Physical Interests; Personality Psychopathology Five (PSY-5): AGGR-r = Aggressiveness–Revised, PSYC-r = Psychoticism–Revised, DISC-r = Disconstraint–Revised, NEGE-r = Negative Emotional-ity/Neuroticism–Revised, INTR-r = Introversion/Low Positive Emotionality–Revised.

Forbey et al. 457

Table 4. Comparison of Collateral Measure Mean Scores for Within Normal Limits (WNL) Group With Content Responsive Invalidity: Over-Reporting (CBIR-OR) and Content Responsive Invalidity: Under-Reporting (CBIR-UR): College Participants.

Criterion measure (total and/or subscale scores)

WNL (n = 877 to

901)

CBIR-OR (n = 96 to

100)

t df p ES (d)

CBIR-UR (n = 57 to 59)

t df p ES (d)Mean Z SD Mean Z SD Mean Z SD

Externalizing Barratt Impulsivity Scale Total −0.09 0.96 0.73 0.91 −8.13 997 <.001 −0.86 −0.58 0.94 3.731 956 <.001 0.51 Drug Abuse Screening Test Total −0.10 0.85 0.65 1.52 −7.62 995 <.001 −0.80 −0.27 0.64 1.437 955 ns 0.19 Michigan Alcohol Screening Test Total −0.09 0.88 0.53 1.45 −6.11 989 <.001 −0.65 −0.33 0.62 2.066 948 ns 0.26 State Anger Inventory Total −0.06 0.96 0.47 1.19 −5.04 990 <.001 −0.54 −0.41 0.74 2.816 951 ns 0.36Internalizing Beck Depression Inventory Total −0.08 0.87 0.85 1.44 −9.07 971 <.001 −0.99 −0.60 0.42 4.497 932 <.001 0.57 Screener for Somatoform Disorders Total −0.06 0.94 0.86 1.17 −9.11 995 <.001 −0.95 −0.71 0.53 5.235 954 <.001 0.68Thought Magical Ideation Scale Total −0.12 0.88 0.99 1.29 −11.41 997 <.001 −1.20 −0.55 0.87 3.591 955 <.001 0.47 Perceptual Aberration Scale Total −0.15 0.78 1.05 1.62 −12.70 999 <.001 −1.33 −0.31 0.99 1.429 958 ns 0.19

Table 5. Comparison of Collateral Measure Mean Scores for Within Normal Limits (WNL) Group With Content Responsive Invalidity: Over-Reporting (CBIR-OR) and Content Responsive Invalidity: Under-Reporting (CRI-UR): Correctional Participants.

Criterion measure (total and/or subscale scores

WNL (n = 391 to

404)CBIR-OR

(n = 37 to 39)

t df p ES (d)

CBIR-UR (n = 157 to 163)

t df pES (d)Mean Z SD Mean Z SD Mean Z SD

Externalizing Barratt Impulsivity Scale Total 0.15 0.88 1.01 0.74 −5.73 439 <.001 −0.99 −0.63 0.93 9.186 559 <.001 0.87 Drug Abuse Screening Test Total 0.08 1.03 0.44 1.03 −2.04 437 ns −0.35 −0.31 0.83 4.278 559 <.001 0.40 Michigan Alcohol Screening Test Total 0.10 1.04 0.31 1.19 −1.18 439 ns −0.20 −0.32 0.76 4.723 559 <.001 0.43 State Anger Inventory Total 0.02 0.81 0.73 1.01 −5.04 427 <.001 −0.86 −0.49 0.68 7.132 550 <.001 0.66Internalizing Beck Depression Inventory Total 0.01 0.80 1.79 1.54 −11.94 431 <.001 −2.00 −0.40 0.82 5.518 551 <.001 0.51 Screener for Somatoform Disorders Total −0.02 0.88 1.54 1.44 −9.77 439 <.001 −1.66 −0.36 0.72 4.385 564 <.001 0.41Thought Magical Ideation Scale Total −0.05 0.85 1.11 1.53 −7.44 440 <.001 −1.25 −0.26 0.93 2.592 562 ns 0.24 Perceptual Aberration Scale Total −0.10 0.19 0.26 0.55 −8.95 438 <.001 −1.49 −0.11 0.21 0.300 563 ns 0.05

458 Assessment 20(4)

For the CBIR-OR response style groups, results indi-cated that the CBIR-OR response style groups demonstrated significantly different mean scores on the majority of the eight measures when compared with the WNL groups after applying a Bonferroni correction. For the college partici-pants, all the externalizing, internalizing, and thought disor-ders measures differed significantly compared with the WNL group (median effect size [d] = −0.91, range = −0.54 to −1.33). For the correctional participants, two of four externalizing, all internalizing, and all thought disorders measures differed significantly compared with the WNL group (median effect size [d] = −1.12, range = −0.20 to −2.00). Finally, for the psychiatric participants, three of four externalizing, both internalizing, and both thought disorders measures were significantly different (median effect size [d] = −1.15, range = −0.27 to −1.84). The median effect sizes include all results (i.e., significant and nonsignificant) in their calculations. For the scales that demonstrated a sta-tistically significant difference across the three samples, inspection of the mean scores indicated the CBIR-OR group reported increased negative functioning on the criterion measures when compared with the WNL group.

Finally, with respect to the CBIR-UR group, after apply-ing a Bonferroni correction, the CBIR-UR response style group demonstrated significantly different mean scores on a

number of the eight measures when compared with the WNL groups. For the college participants, one of four exter-nalizing, all internalizing, and one of two thought disorders measures differed significantly compared with the WNL group (median effect size [d] = 0.42, range = 0.19 to 0.68). For the correctional participants, all externalizing, all inter-nalizing, but no thought disorders measures differed signifi-cantly compared with the WNL group (median effect size [d] = 0.42, range = 0.05 to 0.87). Finally, for the psychiatric participants, two of four of the externalizing, one of two internalizing, and no thought disorders measures were sig-nificantly different (median effect size [d] = 0.64, range = 0.10 to 1.24). The median effect sizes include all results (i.e., significant and nonsignificant) in their calculations. For the scales that demonstrated a statistically significant difference, inspection of the mean scores indicated the CBIR-UR groups reported increased positive functioning or fewer experiences of psychopathology-related symptoms on the criterion measures compared with the WNL groups across the three samples.

DiscussionThe current study examined associations between CBIR response styles, as determined by MMPI-2-RF Validity

Table 6. Comparison of Collateral Measure Mean Scores for Within Normal Limits (WNL) Group With Content Responsive Invalidity: Over-Reporting (CBIR-OR) and Content Responsive Invalidity: Under-Reporting (CBIR-UR): Psychiatric Participants.

Criterion measure (total and/or subscale scores

WNL (n = 94 to

104)CBIR-OR

(n = 38 to 44)

t df p ES (d)

CBIR-UR (n = 15 to 16)

t df p ES (d)Mean Z SD Mean Z SD Mean Z SD

Externalizing Barratt Impulsivity Scale Total 0.14 0.83 1.25 0.93 −7.06 141 <.001 −1.29 −0.88 0.75 4.607 114 <.001 1.24 Drug Abuse Screening Test Total 0.20 1.08 0.71 1.57 −2.26 144 <.001 −0.41 0.09 0.99 .382 116 ns 0.10 Michigan Alcohol Screening Test Total 0.35 1.16 0.67 1.22 −1.52 139 ns −0.27 −0.28 0.89 2.047 112 ns 0.56 State Anger Inventory Total 0.08 0.95 1.15 1.11 −5.87 141 <.001 −1.07 −0.83 0.41 3.725 114 <.001 1.01Internalizing Beck Depression Inventory Total 0.01 0.75 1.57 1.06 −9.54 130 <.001 −1.84 −0.79 0.33 4.159 108 <.001 1.13 Screener for Somatoform Disorders Total 0.05 0.93 1.02 0.92 −5.79 145 <.001 −1.05 −0.59 0.71 2.642 118 ns 0.71Thought Magical Ideation Scale Total −0.19 0.75 0.96 1.28 −6.75 144 <.001 −1.22 −0.57 0.70 1.928 116 ns 0.51 Perceptual Aberration Scale Total −0.17 0.71 1.09 1.46 −7.06 145 <.001 −1.28 −0.47 0.33 1.642 118 ns 0.45

Forbey et al. 459

scale cut scores, and scores on the MMPI-2-RF substantive scales and collateral measures in three groups of partici-pants. As hypothesized, results indicated that individuals identified as potentially engaging in CBIR scored signifi-cantly higher on MMPI-2-RF and collateral measures of dysfunction than did participants who responded to the MMPI-2-RF in a valid manner. Specifically, individuals identified as potentially over-reporting were significantly more likely to report experiencing higher levels of social, behavioral, and psychological difficulties on both the MMPI-2-RF substantive scales and non-MMPI-2-RF col-lateral measures. Those identified as possibly engaging in an under-reporting response style were more likely to report having fewer social, behavioral, and psychological difficulties indicated by their scores on both MMPI-2-RF substantive scales and collateral measures, though the effect sizes of these differences were not as large as those demonstrated for over-reporting. One exception to these trends was that a smaller number of MMPI-2-RF substan-tive scale scores were lower for the under-reporting group in the psychiatric sample. This likely reflects the reduced power available for analyses conducted with this smaller sample.

Most germane to the purpose of the current study, indi-viduals identified as engaging in a potential CBIR response style by MMPI-2-RF Validity scales demonstrated numer-ous statistically significant differences on the eight self-report collateral measures when compared with individuals believed to be responding honestly. This result was demon-strated for collateral measures that were administered either at the same time as the MMPI-2-RF or within a range of 1 to 10 days prior to or afterward (depending on the partici-pant sample). For the CBIR-OR groups, all eight measures for the College participants, six of eight measures for the Correctional participants, and seven of eight measures for the Psychiatric participants were significantly different compared with the WNL group. The pattern of differences observed for these groups indicated that MMPI-2-RF-detected over-reporters endorsed significantly higher levels of internalizing, externalizing, and/or thought dysfunction on the collateral measures. Conversely, for the CBIR-UR groups, of the eight analyzed collateral measures, scores on four measures in the College participants, six of the eight measures in the correctional participants, and three of the measures in the psychiatric participants reflected signifi-cantly lower levels of externalizing, internalizing, and/or thought dysfunction when compared with the WNL group.

Our ability to use three quite different samples in which the collateral measures were administered with different timing points to a robust finding that invalid responding as detected by the MMPI-2-RF validity indicators generalizes to other measures administered along with the test. As noted earlier, this finding has important implications for clinical and correctional assessments in which the MMPI-2-RF is

administered as part of a battery of tests that includes mea-sures that lack validity scales. The implications indicated by validity scale results for interpretation of scores on the MMPI-2-RF substantive scales generalize to other mea-sures and should be applied to their interpretation as well.

Although the current study has a number of strengths, use of “naturally occurring” (rather than simulated) invalid responding, replication in three distinct settings, and vary-ing time frames for the administration of the collateral mea-sures, a number of limitations must be acknowledged as well. Foremost among these is our broad classification of over-reporting or under-reporting based on any elevation on a single validity indicator. In practice, interpretive guide-lines for the MMPI-2-RF indicate a need to consider scores on some validity indicators conjointly. For the present investigation, we opted not to do so in order to first explore the generalizability of validity scale findings at the broadest level. This approach led to roughly 9% of college, 6% of correctional, and 27% of psychiatric participants being identified as potentially engaging in CBIR-OR. Conversely, roughly 6% of college, 27% of correctional, and 10% of psychiatric participants were identified as potentially engaging in CBIR-UR. In examining the comparative CBIR-OR percentages, it is possible that some individuals in the psychiatric group who had legitimate psychological distress were misidentified as over-reporting because they had an elevated score on F-r. On the other hand, when examining the percentages of the CBIR-UR classifications across samples, it is not as likely that the discrepancy between the percentage of correctional and other groups reflected a genuinely overly virtuous or high level of psy-chological adjustment in the correctional participants. Furthermore, in a very limited number of cases, individuals were identified as members of both groups. This finding most likely was the result of using MMPI-2-RF cut scores that were slightly below those of more definitive invalidity as stated in the manual for several of the scales. In any event, although the overall results of the current study are similar to previous findings for the MMPI-2 and MMPI-2-RF substantive scales and collateral measures, future studies might consider using a different method of CBIR group assignment, such as symptom validity testing in com-bination with MMPI-2-RF Validity scale scores.

A second limitation of the current study involved the inability to examine the potential differences in substantive and collateral measure scores that specific MMPI-2-RF over- and/or under-reporting scales might suggest due to relatively small sample sizes for the CBIR-OR and UR groups. For example, individuals who have elevated scores on Fp-r (a measure of over-reporting severe psychopathol-ogy) might have substantially different scores on the MMPI-2-RF substantive scale and/or collateral measures compared with individuals who elevated on Fs (a measure of over-reporting somatic problems). On the other hand,

460 Assessment 20(4)

different under-reporting strategies (as reflected by L-r and K-r) might also lead to differential patters on both the MMPI-2-RF substantive scales as well as collateral mea-sures. We encourage researchers with access to consider-ably larger databases to apply the methodology of the current study to further explore the impact of more specifi-cally delineated response styles.

A third limitation of the current study is that we focused only on self-report collateral measures to explore the impact of CBIR response styles. Therefore, we were not able to examine the potential impact on other information gathered via different means. Future studies should exam-ine whether other forms of client-provided information (e.g., structured and unstructured interviews) might be affected by response biases detected by the MMPI-2-RF validity indicators. A final limitation of the current study involves the fact that only men were examined in the psy-chiatric and correctional groups. Future studies should attempt to include women from these populations as well as to explore the impact of CBIR styles in additional popu-lations with both genders (e.g., other legal settings, employee screening settings, etc.).

The limitations just noted notwithstanding, our findings suggest that if the MMPI-2-RF Validity scale scores iden-tify an individual as potentially engaging in an exaggerated or over-reporting response style, scores on collateral mea-sures administered either conjointly or within a brief period of time are also likely to reflect an attempt to present one’s self in an overly negative or psychologically dysfunctional manner. Although not as strong as the results for the CBIR-OR analyses, if individuals are suspected of under-reporting on the MMPI-2-RF, they are also likely to sup-press scores on collateral measures, reflecting an attempt to present one’s self in an overly positive or psychologically healthy fashion. These findings have both clinical and research implications. Clinically, when the MMPI-2-RF is administered as part of a battery of instruments that includes other self-report scales that lack validity indicators, the cur-rent study suggests that the cautions indicated for MMPI-2-RF interpretation should also be considered when interpreting these collateral measures. In addition, the cur-rent study suggests that researchers who rely on self-report measures that lack validity scales should consider including measures that include such scales, such as the MMPI-2-RF, in their designs.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Yossef S. Ben-Porath is a paid consultant to the MMPI publisher, the University of Minnesota, and distributor, Pearson. As coauthor of the MMPI-2-RF, Dr. Ben-Porath receives royalties on sales of the test.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by a grant to Yossef S. Ben-Porath from the University of Minnesota Press, publisher of the MMPI-2.

Notes

1. The modularized versions of the MMPI-2-CA (computerized adaptive version) involved the administration of selected indi-vidual and/or subsets of MMPI-2 scales, essentially consisting of various abbreviated versions of the test.

2. Whereas the correctional participants were screened for read-ing ability, the same was not true for college students or psy-chiatric participants.

References

American Psychiatric Association. (1994). Diagnostic and statis-tical manual of mental health disorders (4th ed). Washington DC: Author.

Baer, R. A., & Miller, J. (2002). Underreporting of psychopathol-ogy on the MMPI-2: A meta-analytic review. Psychological Assessment, 14, 16-26.

Barratt, E. S. (1985). Impulsiveness subtraits: Arousal and infor-mation processing. In J. T. Spence & C. E Izard (Eds.), Moti-vation, emotion, and personality (Vol. 5, pp. 137-146). New York, NY: North-Holland.

Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 12, 57-62.

Ben-Porath, Y. S. (2013). Self-report inventories: Assessing per-sonality and psychopathology. In J. R. Graham & J. A. Nagl-ieri (Eds.), Handbook of psychology: Vol. 10. Assessment psychology (2nd ed., pp. 622-641). New York, NY: Wiley.

Ben-Porath, Y. S., & Tellegen, A. (2008). MMPI-2-RF (Minnesota Multiphasic Personality Inventory-2 Restructured Form): Manual for administration, scoring, and interpretation. Min-neapolis: University of Minnesota Press.

Bianchini, K. J., Greve, K. W., & Glynn, G. (2005). Review arti-cle: on the diagnosis of malingered pain-related disability: Lessons from cognitive malingering research. Spine Journal, 5, 404-417.

Butcher, J. N., Graham, J. R., Ben-Porath, Y. S., Tellegen, A., Dahlstrom, W. G., & Kaemmer, B. (2001). MMPI-2 (Minne-sota Multiphasic Personality Inventory-2): Manual for admin-istration and scoring (2nd ed.). Minneapolis: University of Minnesota Press.

Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1978). Body-image aberration in schizophrenia. Journal of Abnormal Psy-chology, 87, 399-407.

Cohen, J. (1988). Statistical power analysis for the behavioral sci-ences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Forbey et al. 461

Eckblad, M. L., & Chapman, L. J. (1983). Magical ideation as an indicator of schizotypy. Journal of Consulting and Clinical and Psychology, 51, 215-225.

Forbey, J. D., & Ben-Porath, Y. S. (2007). Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version (MMPI-2-CA). Psychological Assessment, 19, 14-24.

Forbey, J. D., & Lee, T. T. C. (2011). An exploration of the impact of invalid MMPI-2 protocols on collateral self-report measure scores. Journal of Personality Assessment, 93, 556-565.

Garcia, H. A., Franklin, C. L., & Chambliss, J. (2010). Examining MMPI-2 F-family scales in PTSD-diagnosed veterans of Opera-tion Enduring Freedom and Operation Iraqi Freedom. Psycholog-ical Trauma: Theory, Research, Practice, and Policy, 2, 126-129.

Janca, A., Burke, J. D., Isaac, M., Burke, K. C., Costa e Silva, J. A., Acuda, S. W., & Tacchini, G. (1995). The World Health Organi-zation Somatoform Disorders Schedule: A preliminary report on design and reliability. European Psychiatry, 10, 373-378.

Marion, B. E., Sellbom, M., & Bagby, R. M. (2011). The Detec-tion of Feigned Psychiatric Disorders Using the MMPI-2-RF Overreporting Validity Scales: An analog investigation. Psy-chological Injury and Law, 4, 1-12.

Sellbom, M., & Bagby, R. M. (2008). Validity of the MMPI-2-RF (Restructured Form) L-r and K-r scales in detecting under-reporting in clinical and nonclinical samples. Psychological Assessment, 20, 370-376.

Sellbom, M., & Bagby, R. M. (2010). Detection of overreported psychopathological with the MMPI-2-RF form validity scales. Psychological Assessment, 22, 757-767.

Selzer, M. (1971). The Michigan Alcoholism Screening Test: The quest for a new diagnostic instrument. American Journal of Psychiatry, 127, 1653-1658.

Skinner, H. A. (1982). The Drug Abuse Screening Test. Addictive Behaviors, 7, 363-371.

Slick, D., Hopp, G., Strauss, E., & Thompson, G. B. (1997). Vic-toria Symptom Validity Test Version 1.0 professional manual. Odessa, FL: Psychological Assessment Resources.

Spielberger, C. D. (1979). Preliminary manual for the State-Trait Personality Inventory (STPI). Tampa: University of South Florida.

Tellegen, A., & Ben-Porath, Y. S. (2008/2011). MMPI-2-RF (Min-nesota Multiphasic Personality Inventory-2 Restructured Form): Technical manual. Minneapolis: University of Min-nesota Press.

Van der Heijden, P. T., Egger, J. I., & Derksen, J. J. (2010). Comparability of scores on the MMPI-2-RF scales gener-ated with the MMPI-2 and MMPI-2-RF booklets. Jour-nal of Personality Assessment, 92, 254-259. doi:10.1080/ 00223891003670208

World Health Organization. (1992). The ICD-10 international classification of mental and behavioural disorders. Clinical descriptions and diagnostic guidelines. Geneva, Switzerland: Author.

Wygant, D. B., Anderson, J. L., Sellbom, M., Rapier, J. L., All-geier, L. M., & Granacher, R. P. (2011). Association of the MMPI-2 Restructured Form (MMPI-2-RF) Validity Scales with structured malingering criteria. Psychological Injury and Law, 4, 13-23.