a meta-analytic investigation of cognitive ability

8/6/2019 A Meta-Analytic Investigation of Cognitive Ability

1/15

Journal of Applied Psychology1996, Vol. 81, No. 5,459-473 Copyright 1996 by the Ame rican Psychological Association, Inc.0021-90IO/96/S3.00

A Meta-Analytic Investigation of Cognitive Abilityin Employm ent Interview Evaluations: Moderating Characteristicsand Implications for Incremental ValidityAllen I. HuffcuttBradley University Philip L. RothClemson University

Michael A. McDanielUniversity of AkronTh e purpose of this investigation was to explore the extent to which employment in -terview evaluations reflect cognitive ability. A meta-analysis of 49 studies found a cor-rected mean correlation of .40 between interview ratings and ability test scores, suggest-ing that on average about 16% of the variance in intervie w constructs rep resents cognitiveability. Analysis of several design characteristics that could moderate the relationshipbetween interview scores an d ability suggested th at (a) the correlation w ith ability tendsto decrease as the level of structure increases; (b) the type of questions asked can haveconsiderable influence on the magnitude of the correlation with ability; (c) the reflectionof ability in the ratings tends to increase when ability test scores are made available tointerviewers; and (d) the correlation with ability generally is higher for low-complexityjobs. Moreover, results suggest that interview ratings that correlate higher with cognitiveability tend to be better predictors of job performance. Implications for incremental va-lidity are discussed, and recommendations for selection strategies are outlined.

Understanding of the validity of the employment in-terview has increased considerably in recent years. Inparticular, a series of meta-analyses has affirmed that theinterview is generally a much better predictor of perfor-mance than previously thought and is comparable withmany other selection techniques (Huffcutt & Arthur,1994; Marchese & Muchinsky, 1993; McDaniel, Whet-zel, Schmidt, & Maurer, 1994; Wiesner & Cronshaw,1988; Wright, Lichtenfels, & Pursell, 1989). Moreover,these studies have identified several key design character-istics that can improve substantially the validity of theinterview (e.g., structure).However, mu ch less is understood about the constructsassessed in interviews (Harris, 1989; Schuler & Funke,1989). Scattered primary studies have suggested that

Allen I. Huffcutt, Department of Psychology, B radley U ni-versity; Philip L. Roth, Department of Management, ClemsonUniversity; Michael A. McDaniel, Department of Psychology,U niversity of Akron.We thank all of the researchers wh o provided additional in-formation to us about their intervie w studies.Correspondence concerning this article should be addressedto Allen I. Huffcutt, Department of Psychology, B radley U ni-versity, Peoria, Illinois 61625. Electronic mail may be sent viaInternet [email protected].

general factors such as motivation, cognitive ability, andsocial skills may be commonly captured. For example,Landy (1976) factor analyzed ratings on nine separatedimensions from a structured interview an d found threegeneral factors: manifest motivation, comm unication,and personal stability. Campion, Pursell, and Brown(1988 ) found a significan t correlation between interv iewevaluations and a cognitive test battery. Schuler andFunke (1989) found that a multimodal interview that in-cluded vocational, biographical, and situational ques-tions correlated highly with a social skills criterion. Todate, there has been no summary level research in theliterature to assess the extent to which these factors areevaluated and the ir consistency across interview s or theirchange with interview design (e.g., panel format or levelof structure).Understanding the constructs involved is potentiallyimportant. For one thing, there may be overlap betweeninterviews and other selection approaches. The moresimilar the constructs, the greater the possibility that in-terviews may duplicate wh at could be accomplished withless costly paper-and-pencil tests (Dipboye, 1989; Dip-boye & Gaugler, 1993; Harris, 198 9). Furthermore, asHakel (1989) noted, the incremental validity providedby interviews is a key issue in selection. In addition, un -derstanding the constructs involved could lead to general

459


2/15

46 0 HUFFCUTT, ROTH, A ND McDANIELimprovements in interview design, including better rec-ognition of which constructs are most effective for partic-ular jobs. Such improvements could ultimately raise thelevel of validity attainable with interviews.The purpose of this investigation was to explore em -pirically the extent to which employment interview eval-uations reflect cognitive ability. We felt that cognitiveability was a particularly important construct to study inrelation to the interview for two reasons. First, no otherconstruct measures have been shown to predict job per-formance as accurately or as universally. In addition, in-telligence has become a very prominent social issue, asevidenced by publication of Th e Bell Curve (Herrnstein& Murray, 1994). We begin with a conceptual discussionof why one would expect interview evaluations to be sat-urated with cognitive ability. Then, we present and dis-cuss five potential factors that may influence the strengthof this relationship.

Cognitive Ability in Interviewer EvaluationsThere are at least four reasons why interviewer evalua-tions are expected to reflect cognitive ability in a typicalinterview. First, intelligence may be one of a small num-ber of issues that are highly salient in many interview sit-uations. Research suggests that interviewers tend to basetheir judgments on a limited number of factors (Roth &Campion, 1992; Valenzi & Andrews, 1973). This is notsurprising given general limitations in human informa-tion processing (see Solso, 1991). Moreover, it appears

that the judgments behavioral observers make are typi-cally guided by overall impressions (Kinicki & Lock-wood, 1985;Srull&Wyer, 1989). Thus, many interview-ers may be focusing on a limited number of generalthemes, such as whether the applicant has the appropri-ate background, can fit in with other employees, and isbright enough to learn the job requirements quickly. Inturn, their general impressions along these themes arelikely to have considerable influence on the finalevaluations.Second, applicants with greater cognitive ability maybe able to present themselves in a better light than appli-cants with lower cognitive ability. Applicants clearly en-gage in impression management behaviors (Gilmore &Ferris, 1989), using techniques such as ingratiation, in-timidation, self-promotion, exemplification, and suppli-cation (Jones & Pittman, 1982; Tedeschi & Melburg,1984). Those with greater cognitive ability may be betterat knowing which strategies are most likely to succeed inthat situation and when to back off from such strategies(see Barren, 1989). At a more general level, the link be-tween impression management and cognitive ability ishighlighted in a relatively new theory of intelligence (seeGardner & Hatch, 1989). In his theory, Gardner main-

tains that interpersonal skills, namely the capacity to dis-cern and respond appropriately to other people, are oneform of intelligence.Third, at least some of the questions commonly askedin employment interviews could elicit ability-loaded re-sponses. For example, questions of a more technical na-ture are likely to be answered more effectively by appli-cants with higher cognitive ability. These applicantsprobably are able to think in more complex ways andhave a greater base of retained knowledge from whichto work. Abstract questions may also be answered moreeffectively by applicants with higher cognitive ability. Forexample, two of the most frequently asked questions are,"What do you consider your greatest strengths and weak-nesses?" and "What [college or high school] subjects didyou like best and least?" (see Bolles, 1995). More intelli-gent applicants may be better at thinking through suchquestions andgiving more desirable responses.Fourth, cognitive ability may be indirectly capturedthrough background characteristics. Intelligence, moreso than any other measurable human trait, is strongly re-lated to many important educational, occupational, eco-nomic, and social outcomes ("Mainstream science,"1995). Thus, on average, more intelligent people arelikely to have more and better education, greater socialand economic status, an d better previous employment.Such information, whether it is reviewed before the in-terview or emerges during the interview, could influenceinterviewers' ratings in a favorable manner. Fo r example,researchers (Dipboye, 1989; Phillips & Dipboye, 1989)

have found that preinterview information can stronglyinfluence both the interview process and subsequent rat-ings. In general, the more influence background informa-tion has on the ratings, the greater the chance that theseratings will reflect cognitive ability.In summary, it appears that a number of mechanismsby which interview ratings can become saturated withcognitive ability. One of these mechanisms, interviewerevaluation of applicants' ability to learn job require-ments quickly, represents a relatively direct measure-ment of the ability construct. Each of the other threemechanisms represents more of an indirect influencefrom ability. In particular, cognitive ability influences ap-plicant behavior during the interview, generation of re-sponses, and background characteristics, all of which inturn influence the final ratings. It should also be notedthat these four mechanisms are not mutually exclusive inthat more than one may be operating in a given situation.Potential Moderators of the Interview-Ability Correlation

The level of structure is a widely researched character-istic of interview design. The two most prominent aspects


3/15


4/15

462 HUFFCUTT, ROTH, AND MeDANIELcording to the comp lexity of the job for which the appli-cants are applying. Jobs differ widely in complexity, andjobs of greater complexity generally require a higher levelof mental skills ("Mainstream science," 1995). Such atenet is supported empirically by the finding that the va-lidity of ability tests tends to increase with the level ofcomplexity (Gandy, 1986; Hunter & Hunter, 1984; Mc-Daniel, 1986). Therefore, it is posited in an interviewthat the interviewers recognize the increased necessity forsuch skills with more complex jobs, and they place moreemphasis on their assessments. Such a tenet assumes thatinterviewers not only recognize a job as being of highcomplexity but also successfully incorporate ability intotheir assessments. In general, we predicted that the corre-lation with ability would be greater with interviews formore complexjobs.Finally, there may be an associationbetween the mag-nitude of the interview-ability correlation and the mag-nitude of the validity coefficient (i.e., the correlation be-tween interview ratings and job performance). Researchsuggests that cognitive ability is a strong and consistentpredictor of job performance (Hunter & Hunter, 1984).Accordingly, it can be argued that the more saturated theratings are with ability, the higher the resulting validity ofthose ratings is likely to be. Alternately, it can be said thatinterviews become more valid when they capture cogni-tive ability. Thus, we predicted that in general, ratingsthat correlate more highly with cognitive ability shouldbe morevalid predictorsof job performance. Such apre-diction is relevant to all jobsregardless of complexity be-cause cognitive ability is still a valid predictor, even forlow-complexity jobs.

MethodSearch for Primary Data

We conducted an extensive search for interview studies thatreported a correlation between interview ratings and some typeof cognitive ability test. Datasets from previous meta-analyseswere prime sources for locating studies (Huffcutt & Arthur,1994; McDaniel et al., 1994; Wiesner & Cronshaw, 1988). Sup-plemental inquiries were also made of prominent researchers inthe interview area in order to obtain any additional studies notincluded in the above datasets.Tw o main criteria were used in deciding which of the studiesreporting an interview-ability correlation would be retained.First, the interview had to represent a typical employment in-terview. Eight studies did not meet this criteria and were ex-cluded. Three of these involved a procedure known as an ex-tended interview, where the interview is combined with severalassessment center exercises (Handyside & Duncan, 1954; Tran-kell, 1959; Vernon, 1950). Twostudies used objective biograph-ical checklists rather than true interviews (Distefano & Pryer,1987; Lopez, 1966). One interview was designed deliberately toinduce stress (Freeman, Manson, Katzoff, & Pathman, 1942).

Finally, in two studies interviews were used as an alternatemethod to assess job proficiency (Hedge & Teachout, 1992;Ree, Earles, & Teachout, 1994). Second, a study had to providesufficient information to allow coding on a majority of the fivemoderator characteristics. Three studies did not report suffi-cient information and were dropped (Conrad & Satter, 1945;Darany, 1971;Friedland, 1973). In total, we were able to locate49 usable studies after application of the above decision rules,with a total sample size of 12,037.

Such a dataset isnotablegiven the general difficulty in findingstudies that report correlations among predictors (Hunter &Hunter, 1984). These studies represented a wide range of jobtypes, organizations, subjects, and interview designs. Sourcesfor the studies were similarly diverse and included includingjournals, unpublished studies, technical reports, and dissert-ations. Thus, we were reasonably confident that these studiesrepresented a broad sampling of employment interviews.

As expected, there wasconsiderable diversity amongthe abil-ity measures to which interview ratings were correlated. Tobet-ter understand the ability measures used, we compiled somesummary statistics. Of the 49 studies in our dataset, 11 (22.4%)used a composite test such as the Wonderlic Personnel Test(Wonderlic, 1983) where various types ofability-loading ques-tions (e.g., math, verbal, and spatial) are combined into onetest. In 31 of the studies (63.3%), separate subtestsofindividualfactors were administered and these scores were then combinedto form a composite. Lastly, in 7 of the studies (14.3%), sepa-rate subtests of individual factors were administered, but noability composite wasformed.

In general, we felt that the first two categories of tests listedabove were all reasonable (albeit not perfect) measures ofgen-eral cognitive ability. In the first category, the test itself was acomposite of individual factors, and in the second category, acomposite was formed from individual subtests. We had someconcern about the third category because no composite wasformed. However, eight studies from the second category re-ported ability correlations with individual factors, as well aswith the composite ability measure. Our analysis of these eightstudies suggested that the highest individual correlation was afairly accurate estimate of the composite correlation. In partic-ular, the highest individual correlation from these studies corre-lated .98 (p < .0001) with the composite correlation. Thus, wetook the highest individual correlation in these seven studies.

Coding of Study CharacteristicsLevel of interview structure was coded with a variation of

the framework developed by HufFcutt and Arthur (1994). Theyidentified four progressively higher levels of question standard-ization, which Conway et al. (1995) later expanded to five. Theyalso identified three progressively higher levels of response eval-uation. We combined various combinations ofthese twoaspectsof structure into three overall levels corresponding to low,me-dium, and high. Studies were classified as lowstructure if therewere no constraints or very limited constraints on the questionsand the evaluation criteria. Studies were classified as mediumstructure if there were a higher level of constraints on the ques-tions and responses were evaluated along a set of clearly defineddimensions. Finally, studies were classified as high structure if


5/15

INTERVIEW CONSTRUCTS 463there w ere precise specifications in the wording and in the num-ber of questions without variation or with very limited flexibil-ity to choose questions and probe, and the responses were eval-uated individually by question or along mu ltiple dimen sions.We also attempted to code both m edium- and high-structureinterviews by type of question or content. Studies were classifiedas situational if most or all of the questions involved presenta-tion of job-related scenarios to which the applicants indicatedhow they wou ld respond. Similarly, studies were coded as be -havior description' if most or all of the questions involve d ask-ing applicants to describe actual situations from their past thatwould be relevant to the job for which they were applying. Al-though a number of studies clearly fell into one of these twocategories, there were two studies that used more th an one typeof question. Campion, Campion, and Hudson (199 4) had botha situational section and a past behavior section. Furthermore,Campion et al. (1988) used a combination of situational, jobknowledge, job simulation, and worker requirement questions.In the fo rme r case, w e coded the two sections as separate studiesbecause the two types of questions were not mixed (i.e., on esection was given an d then the other), and separate informationon the correlation with ability wa s provided. In the latter case,the question types were mixed, an d separate information wa snot provided. Harris (1989) denoted this mixture of four ques-tion types as a comprehensive structured interview. We codedthis study as comprehensive but did not analyze it as a separateconten t category because there was only one of its type.Although it is obviously possible for studies that use a partic-ular type (or a combination) of questions to be of mediumstructure, most tend to be of high structure. The most likelyexception is with behavior description studies because the de-sign allows for considerable interviewer discretion (Janz,1982). In our dataset, all of the studies for which we could makea content classification were of high structure, including the be-havior description ones. Consequently, for the medium-struc-ture studies, we attempte d to m ake a simple dichotomou s clas-sification. Specifically, studies were coded as to whe ther at leastsome technical, problem-solving, or abstract reasoning (e.g.,situational) questions were systematically included .In effect, content was a nested variable operating under struc-ture in this investigation, in that it had different categories atdifferent levels of structure (see K eppel, 1991, for a discussionof nested variables.) Such nesting did not present a problembecause ou r initial hypothesis was that differences in ability sat-uration would emerge when interviews at variou s levelsof struc-ture are further broken down by content. No distinction ofcontent was made with low-structure interviews because infor-mation regarding the types of questions asked by the interview-ers was generally not provided.Availability ofability test information wascoded as a dichot-omy. Specifically, studies were coded as to whether interviewershad access to cognitive ability test scores at an y time du ring theinterview process.Job complexity wa s coded with a three-level framework de -veloped by Hunter, Schmidt, an d Judiesch (199 0). This frame-work is based on ratings of "Data and Things" from the Dictio-nary of Occupational Titles (U.S. Department of Labor, 1977)and is a modified version of H unter's (1980) original job com-plexity system. Specifically, unskilled or semiskilled jobs such

as truc k driver, assembler, and file clerk were coded as low com-plexity. Skilled crafts, technician jobs, first-line supervisors,lower level administrators, and other similar jobs were coded asmedium complexity. Finally, jobs such as managerial, profes-sional, and those involving complex technical set-up werecoded as high complexity. As Gandy (1986) noted, job com-plexity classifications essentially reflect the information-pro-cessing requirem ents of a position, an d they do not capture thecomplexities relating to interactions with people.The validity coefficient of the interview was recorded as re-ported in the studies and were uncorrected for artifacts. We in-cluded only coefficients involving job performance criteria be-cause mixin g performa nce and training criteria did no t seemappropriate, an d there were too few training coefficients to do aseparate analysis. Also, coefficients representing overall ratingson both the interview and performance criteria were preferred.If not presented, the coefficients for the individual dimensionswere averaged.In cases where multiple pe rformance evaluationswere made, typica lly in situations where more tha n on e ap-praisal instrument wa s used, the resulting validity coefficientswere averaged.Whereas the above five factors constituted the independent(i.e., mo derato r) variables, the dependent variable in this inves-tigation was the degree to which ability wasreflected in the in-terview ratings. We recorded the observed (uncorrected) corre-lation between interview ratings and ability test scores. Prefer-ence was given to correlations involving overall interviewratings rather than individual dimensions and to correlationsinvolving a composite ability score rather than individual abil-ity factors. As noted above, when correlations were reportedonly for individua l ability factors, we took the highest individua lcorrelation as an estimate of the correlation with the abilitycomposite.

As expected, some of the studies did not report en ough infor-mation to make a complete coding on all of the above charac-teristics. In these cases, we made a concerted attempt to contactthe authors directly to obtain further information. Althoughthis was not always possible, we did manage to reach a numberof them and were able to mak e additional codings. In total, wewere able to code all 49 (100% ) of the studies for level of struc-1 We intended the behavior description label to be somewhatgeneral. Janz (1982) actually called hi s original format the pat-terned behavioral description interview because interviewerscould choose selectively from patterns of questions establishedfor each dimension and probe applicant responses freely. Sev-eral later studies used the same type of question but w ith higherlevels of constraint, and renaming the format in the process.For example, Pulakos and Schmitt (1995) and Motowidlo et al.(1992) comp letely standardized the questions, with the formerbeing called an experienced-based interview and the latter beingcalled a structured behavioral interview. The key factor for clas-sification in our behavior description category was not the ex-tent of constraints (i.e., medium versus high structure), butrather that the questions requested information about past realsituations that would be relevant to the job. However, in thismeta-analysis, all of the studies with situational behavior de -scription and comprehensive content were of high structure.


6/15

46 4 HUFFCUTT, ROTH, AND Me DANIELture, 46 ( 9 4 % ) for availability of ability scores, 49 (100% ) forjob complexity, 41 (84%) for the validity coefficient, and 49(100%) for the interview-ability correlation. Regardingcontent, we were able to code 18 high-structure studies as hav-ing situational, behavior description, orcomprehensive content,and 19 medium-structure studies as containing some or no de-liberate cognitive content. Thus, in total wewere able to code37 studies for content ( 7 6% ) . Overall, these results indicate thatwith the additional information obtained, we were able to codemost studies on most variables.To ensure accuracy of coding, we independently coded everystudy on the above characteristics. Differences were then inves-tigated and resolved by consensus. In select cases, the authorsof the study were contacted for verification. Th e correlationsbetween our initial set of ratings (before consensus resolution)were .90 for structure, .71 for availability of ability scores, 1.00for interview content, .89 for job complexity, .98 for the validitycoefficient, and .95 for the interview-ability correlation (p

a meta-analytic investigation of cognitive ability

Documents