evaluations of tests used for making selection and promotion decisions

8/6/2019 Evaluations of Tests Used for Making Selection and Promotion Decisions

1/11

24 INTERNATlONAL IOURNAL OF SELECTION A N D ASSESSMENT

Evaluations of Tests Used fo rMaking Selection and PromotionDecisionsDavid A. Kravitz, Veronica Stinson and Tracy L. ChavezParticipants(N= 212) ated the fairness,job relevance, appropriateness, and invasiveness of16 tests that could be used to select o r promote people into production or managementpositions. Fairness, job relevance and appropriateness were highly correlated, and werecombined to form a composite evaluation scale. Evaluations and invasiveness ratings variedamong the 16 tests, with the most positive ratings given to interviews and work samples andthe most negative ratings given to astrology, graphology and polygraphs. Evaluations of fourtests were affected by the position (manager vs. production worker). Evaluations of 11 testsand invasiveness ratings of two ests were affected by respondent experience with the test.Respondents who had experienced the tests evaluated them more positively and consideredthem to entail a smaller invasion of privacy. Responses were not affected by whether the testwas to be used fo r selection versus promotion decisions.

David A. Kravitz, VeronicaS t i s o n and Tracy L. Chavez,Department of Psychology,Florida International Univer-sity, North Miami Campus,3000 NE 14 5 Street, NorthMiami, L 33183-3600,USA.

For correspondence- re-sent address 2079 NE I96Terrace, North Miami Beach,FL 33179, USA.

Volume 4 Number 1

n employees first important contact withA an organization typically occurs when heor she applies for a job. Because first impressionsstrongly influence future relations (Rynes, Hene-man and Schwab 1980; Zunin and Zunin 1972),the employees impression of the selectionprocedure may have long-term implications.Fo r example, reactions to selection proceduresmay affect applicant attraction to the organiza-tion, motivation to join and work diligently forthe organization, tendency to legally challengeorganizational decisions and actions, willingnessto recommend the organization to others,commitment to the organization and perfor-mance, as well as the validity and utility of theselection procedures themselves ( h e y 1992;Folger and Cropanzano in press; Gilliland andHonig 1994; Macan, Avedon, Paese and Smith1994; Murphy 1986; Robertson, Iles, Grattonand Sharpley 1991; Smither, Reilly, Millsap,Pearlman and Stoffey 1993). In short, organiza-tional climate, performance and profits can beimpaired if applicants react negatively to theselection procedures. Consistent with this point,Sdunitt and Gilliland (1992) call for research onapplicant reactions to selection processes.Reactions to selection procedures may varyalong many dimensions, and these dimensionswill be more or less strongly associated withoverall evaluations. The dimension most com-monly assessed is perceived job relevancelfacevalidity (Fraser and Anderson 1989; Gilliland1994; Macan ef aZ. 1994; Robertson ef al. 1991;Rynes et al. 1980; Smither ef al. 1993). This

January 1996

research clearly reveals that job relevance affectsevaluations of selection procedures and subse-quent reactions (e.g. organizational commitment)and behaviours (e.g. performance).A second important dimension is perceivedfairness of the selection test. During the past 15years, research has found that perceptions offairness affect reactions to many organizationaldecisions (eg. Brockner 1990; Folger and Green-berg 1985; Greenberg 1986, 1990; Konovskyand Cropanzano 1991; McFarlin and Sweeney1992; Nacoste 1987; Sheppard and Lewicki1987). In addition, Cropanzano has pointed ou tthat valid selection tests are often consideredunfair, and that tests considered fair may beinvalid (Cropanzano and Hunsberger 1994;Folger and Cropanzano in press). Cropanzanoand his colleagues refer to this inconsistency asthe justice dilemma. The justice dilemma makes itdifficult for human resource managers to developvalid selection procedures that applicants willconsider fair. This dilemma could be resolved byreplacing unfair selection tests with equally validtests that are considered fair, but this requiresinformation about the perceived fairness of selec-tion tests. In short, both the justice dilemma andresearch on fairness in other domains highlightthe need for research on perceived fairness ofselection tests. Such work has recently beenreported (Gilliland 1994; Gilliland and Honig1994; Kluger and Rothstein 1993; Macan et al.1994). This work has provided some informationabout which tests are considered fair, and hasshown that perceptions of fairness are positively

BBkckwell Publishers Ltd. 1996, 108 Cowley Road Oxford OX4 IJF,UK and238 Main Street, Cambridge.MA 02142, USA.


2/11

EVALUATlONS OF TESTS 25

associated with more general affective andbehavioural reactions.For some time, research on drug testing,honesty testing and polygraphs has dealt withthe invasion of privacy (Garland, Giacobbe andFrench 1989; Karren 1989; Moore and Stewart1989; Stone and Kotch 1989) and more recentwork has assessed invasiveness of other selectionprocedures as well. This work has found thatselection procedures vary in invasiveness, andthat invasiveness is associated with more generalevaluations (Kluger and Rothstein 1993; Rosse,Miller and Stecher 1994; Stone, Stone and Hyatt1989).Finally, perceptions of job relevance, fairnessand invasiveness should determine judgementsof appropriateness. That is, applicants shouldconsider it appropriate for organizations to usetests that are job relevant, fair and non-invasive,and inappropriate for organizations to use teststhat are irrelevant, unfair and invasive. Thus,appropriateness should represent a generalevaluative judgement, and should vary withmore basic judgements of job relevance, fairnessand invasiveness.In summary, research on applicant reactions toselection tests has focused on the dimensions ofjob relevance, fairness and invasiveness. Aconceptualization that ties these three variablestogether, along with a number of other variables,has been provided by Gilliland (1993).Gillilandsjustice model of applicant reactions to selectionsystems posits that selection practices, includingchoice of selection test, will affect applicantperceptions of whether rules of procedural anddistributive justice are being met. These rulesinclude job relatedness and invasiveness. Percep-tions of rule satisfaction, in turn, will affectjudgements of overall fairness. These fairnessjudgements will influence more general evalua-tions and reactions.Both theoretical and empirical work clearlysuggest that perceptions of the job relevance,fairness and invasiveness of selection tests willstrongly affect more general evaluations of thetests. The first two hypotheses are consistentwith this previous work, and especially withGillilands (1993) model:

Hypothesis 1: The type of selection test willinfluence perceptions of job relevance and inva-siveness.Hypofhesis 2: Job relevance and invasiveness willbe associated with perceived fairness of the test.Gilliland (1993) hypothesized that priorexperience with a selection procedure wouldincrease the salience of procedural rules asso-ciated with those experiences. This implies thatexperience will lead to more extreme reactionsto selection bests- ither positive or negative

depending on whether the test violates orcomplies with the procedural justice rules. Onthe other hand, experience also serves toeliminate misperceptions, and thus the effect ofexperience may depend on whether pre-existingstereotypes of the selection test are more or lessnegative than the reality. Both of theseperspectives suggest that experience may differ-entially affect reactions to different selectiontests. Consistent with this suggestion, Fruhner,Schuler, Funke and Moser (1991, as cited inSchuler 1993) found that experience increasedevaluations of interviews but not of psycho-logical tests. Finally, other research indicates thatfamiliarity leads to liking (Bornstein 1989;Zajonc 1968), so experience may result in morepositive evaluations of all selection tests. It is notclear which of these processes will be mostimportant, so one purpose of the present studywas to obtain descriptive information about theeffects of experience with specific selection testson evaluations of those tests. Thus, the thirdhypothesis is nondirectional:

Hypothesis 3: Reactions to selection tests will beinfluenced by the respondents experiences withthe tests. This influence may be constant acrosstests or it may vary with the test.Thornton (1993) pointed out that reactions toselection tests may vary with job level, andcalled for research to test this moderating effect.The authors know of no such research, so in thepresent study the effect of job level on reactionsto selection tests is assessed. In general, it isexpected that selection tests that focus ontechnical skills or abilities will be consideredmore appropriate for production workers thanfor managers, and tests that focus on personalitytraits or general behaviour will be consideredmore appropriate for managers than for produc-tion workers. However, these expectations aretentative, so the hypothesis is nondirectional:Hypothesis 4: Reactions to selection tests will beinfluenced differentially by the position beingfilled.Finally, most selection tests could also be usedfor making promotion decisions. Indeed, the useof internal recruitment implies that selection andpromotion can sometimes be seen as identical(Bureau of National Affairs 1988). Alternatively,because internal applicants are known to theorganization one might argue that promotionsshould be based solely on prior performancewithin the organization, with no other test beingappropriate. Thus, the present study addressesreactions to the use of tests for two differentdecisions- election and promotion. The finalhypothesis refers to the moderating effect of thedecision:

0 Blakwell Publishers Ltd. 1996 Volume 4 Number 1 January 1996


3/11

26 INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT

Hypothesis 5: Reactions to selection tests will beinfluenced by the decision (selection versuspromotion) for which they are being used. Thisinfluence may be constant across tests or it mayvary with the test.In summary, previous research has revealedthat reactions to different selection tests vary

along such dimensions as job relevance, fairnessand invasiveness. In the present study subjectsevaluated 16 tests on four dimensions - obrelevance, invasiveness, fairness and overallappropriateness. To test for moderating effects,position and decision were manipulated andexperience with the test was assessed. Thisresearch extends previous w ork in several ways.First, reactions to 16 different selection tests areobtained. With a few exceptions (Fraser andAnderson 1989; Rynes and Connerley 1993;Schuler 1993; Smither e f al. 1993; Stone ek al .1989) previous studies have included no morethan four tests at a time. Second, simultaneousassessment is made of job relevance, fairness andinvasiveness; only Kluger and Rothstein (19 93)have included all three of these key variables.Third, as suggested by Thornton (1993). themoderating effect of job level is assessed. Fourth,the moderating effect of decision (selection vs.promotion) is assessed. The authors d o no t knowof any other research on the moderating effectsof job level or decision. Finally, the moderatingeffect of experience with the test is assessed.Gilliland (1993) suggests that experience maymoderate reactions to selection systems, butonly Fruhner ef al. (1991, as cited in Schuler1993) have actually examined the effects ofexpe rience. Resu lts of the present re search willfill the gaps in the literature as described above,These results also will have implications forGillilands (1993) justice model of applicantreaction to selections tests, and may be usefulfor resolving the justice dilemm a.

MethodParticipantsParticipants were 212 students at Florida Inter-national University, a m id-sized urban universitywith two campuses in the greater Miami area.They received class credit for participation inthis study. Nine participants with more than on epiece of invalid data in their evaluations of thetests were dropped from the sample. Fourteenparticipants with a single piece of invalid datawere retained in the sample, so sample sizevaries slightly across analyses. Cell size rangedfrom 31 t o 36 participants.The final sample included 52 males and 148females. The sample was ethnically diverse: itincluded 77 White non-Hispanics, 24 Black non-

Hispanics, 88 Wh ite H ispanics, and 14 people ofother ethnicities. The highest level of educationcompleted included high school (N = 86).A.A.college degree (N = 107), B.A. or B.S. collegedegree (N = 8 ) and graduate degree (N = 2 ) .The respondents ranged in age from 1 7 to 45(Mdn = 2 I) and had m ore work experience thantraditional college students. They had workedfor 0-27 years (Mdn = 3) at 0-14 jobs (Mdn =3) and had been promoted 0-20 times (Mdn =0;M = 1.4). They currently worked (r-64 hoursper week (Mdn = 22). Sixteen respondentsreported that they were not currently working.The most common jobs reported were student(N = 60), salesperson (N= 39),professional (N= 13),service worker (N = 12) and first-levelmanager (N = 12). It is clear that mostrespondents who reported their job as studentwere actually working part-time at some otherjob; this apparent inconsistency resulted from apoorly written question about current job.An experienced subsample was created byselecting respondents on the basis of their jobexperience. All 74 participants in this subsamplehad been promoted at least once (M = 2.8) andwere currently working at least part-time (M =29.8 h). The y had worked for a m ean of 7.2 yearsa t a mean of 4.9 jobs. All analyses were repeatedon this experienced subsample, and results ofthese supplementary analyses are reported whenthey differ meaningfully from results of theprimary analyses.

Design, materials and procedurePosition and decision type were manipulatedbetween-subjects, and test ty pe was m anipulatedwithin-subjects. The position w as skilled produc-tion worker, second-level manager o r therespondents current position. The respondentscurrent position was included for exploratorypurposes and to enhance generalizability of theresults. The decision was either selection (hiringa person into the given position) or promotion(promoting an employee one step into the givenposition). The questionnaire included the follow-ing 16 tests: interview, reference letters, accom-plishment record, job skills test, work sample,drug tests, criminal record, cognitive ability test,personality test, medical examination, poly-graph, paper-and-pencil honesty test, physicalability test, mental health test, graphology, andastrology .Dozens of selection tests could have beenincluded in this research. To maximize theacquisition of information, the authors wantedto include as many tests as possible. However,the use of too many tests would result in aquestionnaire of inordinate leng th. The final setof 16 tests were selected on the basis of aliterature review, and the tests were intended to

Volume 4 Number 1 January 1996 0 Blackwen Publishers Ltd. 1996


4/11

EVALUATIONS OF TESTS 27

vary in fairness, frequency of use, legality,validity, job relevance and intrusiveness.The questionnaire began with a page ofinstructions that explained the task and guaran-teed anonymity. Each of the 16 subsequentpages briefly described one of the tests and thenasked the respondent to rate the test with regardto fairness, relevance, invasion of privacy andoverall appropriateness. (Copies of these testdescriptions can be obtained from the firstauthor.) Fairness, job relevance and invasivenesswere included to replicate previous research; inaddition, they are incorporated in Gillilands(1993) model. Appropriateness was includedbecause i t is a more general judgement, andthus ratings of appropriateness should dependon perceptions of fairness, job relevance andinvasiveness. Ratings were given on nine-pointLikert-format response scales with endpoints ofstrongly disagree and stongly agree. Due to thelarge number of selection tests the authors wereforced to use single-item measures of theseconcepts. The items were as follows. Fairness: Itis fair for Company X to use [selectionprocedure] when selecting [position]. Job rele-vance: Information provided by [selection pro-cedure] is related to the ability to perform well asa [position]. Invasiveness: Using [selection pro-cedure] would be an invasion of privacy.Appropriateness: Overall, it is appropriate forCompany X to use [selection procedure] todecide who to [decision] as a [position].Respondents were then asked whether the testhad been used when they had previously appliedfor jobs or promotions. Possible responses were:yes, no, dont know, dont remember, have neverapplied for a job/promotion. Ten differentrandom orders of these 16 pages were used,and order was treated as a blocking factor. Onthe last two pages of the questionnaire therespondent was asked to report h idher gender,ethnicity, education, current job, years in theworkforce, number of previous jobs, hoursworked per week, and number of promotions.

Responses were given on scan sheets. The entiresession typically took less than 40 min.

ResultsCorrelafions among dependent variables and scaledevelopmentCorrelations among the four ratings are given inTable 1.Consistent with Hypothesis 2, ratings offairness were strongly associated with judge-ments of relevance and invasiveness. Consistentwith the greater emphasis on job relevanceobserved in the empirical literature and inGillilands (1 93) model, fairness was moreclosely related to job relevance than toinvasiveness. Similarly, appropriateness washighly correlated with fairness and job relevance.Correlations between these three variables andinvasiveness were weaker but still substantial.The high intercorrelations among ratings offairness, relevance and appropriateness implythat these measures represented a single con-struct. Thus, responses to these three items werecombined to provide a measure of overallevaluation. This approach is consistent withprevious research (Fraser and Anderson 1989;Kluger and Rothstein 1993; Robertson et al.1991). Internal consistency reliabilities (Cron-bachs alpha) of this three-item scale ranged from0.72 to 0.92 when computed for each of the 16tests. Invasiveness was not included in thisevaluation scale because doing so woulddecrease the reliabilities by a mean of 0.07(and up to 0.35). All subsequent analyses wereperformed on this measure of overall evaluationand on the invasiveness ratings.Multivariate analyses of variance on ratings

Evaluation scale. A Position (2)x Decision (2)X Test Type (16) multivariate analysis ofvariance (MANOVA) with repeated measureson the third factor was performed on the

Table 1: Correlations among dependent variables

! Vrriable Fairness Relevance Appropriateness Invasiveness IFairness 0.95 1.00 -0.78Relevance 0.70 0.96 -0.74Appropriateness 0.73 0.72 - .80Invasiveness - .40 -0.33 -0.36Note: High values indicate high levels of fairness, job relevance, appropriateness, andinvasiveness. Correlation coefficients above the diagonal were obtained by averaging ratingsover participants for each test, and then computing correlations over the 16 tests. Correlationcoefficients below the diagonal were obtained by computing correlations over participantsfor each test, and then averaging the coefficients across the 16 tests using Fishers r toZ transformation.

Volume 4 Number 1 January 1996BlackweU Publisherr Ltd 1996


5/11

28 INTERNATIONAL JOU RNAL OF SELECTION AND ASSESSMEhlT-~~Table 2: Mean evaluations of tests

TestPositiortb Experienc6 wi thMeart" the test

S Pw SLM Yes N o~ ~ ~~

InterviewWork sampleAccomplishment recordJob skills testDrug testReferencesCognitive ability testMental health assessmentCriminal recordHonesty testPersonality testMedical examinationPhysical ability testPolygraphGraphologyAstrology

7.6,?.3&6.9k6.8bc6.6h6.4cd5.9de5 . h5.4,fg5.&,4.8,4.8,4.7,3.22.01.3

7.47.56.76.76.56.25.65.44.84.64.14.95.93.21.91 . 3

7.77.37.27.36.66.66.35.95.8**6.0*5.3**4.84.0**3.62.21.3

7.87.97.47.66.86.97.07.15 .36.06.16.55.94.5

7.4*7.1*6.7**6.6**6.56.1**5.7**5.5*5.45.04.6**4.6**4.6*3.1*

Nofe: High values indicate that use of the test is positively evaluated'&Means with common subscripts do not differ a t the p < 0.001 level by paired t-tests. These means andcomparisons are based on data from all three position conditions.I' SPW, Skilled production worker; SLM, second-level manager.only one participant had experienced the latter.p < 0.05, * * y < 0.01.No means are presented for graphology or astrology because nobody had experienced th e former and

evaluation scale. The own posifion condition wasexcluded from this analysis because the varietyof jobs held by the respondents makes itimpossible to interpret results including thiscondition. Significant effects were obtained fortest type, Wilks F (15, 117) = 177.14, p

evaluations of tests used for making selection and promotion decisions

Documents