reasoning tests technical manual

INTERNATIONAL

general,critical &graduateTEST BATTERYtechnical manual

ONTENTS1 THEORETICAL OVERVIEW

2 THE GRADUATE & GENERAL REASONING TESTS (GRT1 & GRT2)

3 THE CRITICAL REASONING TEST BATTERY (CRTB)

4 THE PSYCHOMETRIC PROPERTIES OF THE REASONING TESTS

5 ADMINISTRATION INSTRUCTIONS

6 REFERENCES

c

31 GRT2 STANDARDISATION SAMPLES

2 GENDER DIFFERENCES ON GRT23 CORRELATIONS BETWEEN GRT2 AND AGE

4 GENDER DIFFERENCES ON GRT15 RELATIONSHIP BETWEEN AGE AND GRT16 GENDER DIFFERENCES ON CRTB

7 RELATIONSHIP BETWEEN AGE AND CRTB

8 COEFFICIENT ALPHA FOR GRT2 SUB-SCALES

9 COEFFICIENT ALPHA FOR GRT1 SUB-SCALES

10 TEST-RETEST RELIABILITY ESTIMATES FOR GRT111 COEFFICIENT ALPHA FOR CRTB SUB-SCALES

12 DIFFERENCES BETWEEN COMPUTER VS CONVENTIONAL METHODS OF TEST

ADMINISTRATION

13 CORRELATIONS BETWEEN THE GRT1 SUB-SCALES

14 CORRELATIONS BETWEEN THE GRT2 SUB-SCALES

15 CORRELATIONS BETWEEN CRTB VERBAL & NUMERICAL

16 CORRELATIONS BETWEEN THE GRT1 AND AH5 SUB-SCALES

17 CORRELATIONS BETWEEN THE GRT2 AND AH3 SUB-SCALES

18 CORRELATIONS OF GRT2 WITH TECHNICAL TEST BATTERY

19 CORRELATIONS OF GRT2 WITH CLERICAL TEST BATTERY

20 CORRELATIONS BETWEEN THE EXPERIMENTAL VERSIONS OF THE VCR1 & NCR1AND THE VERBAL NUMERICAL SUB-SCALE OF THE AH5

21 CORRELATIONS BETWEEN VCR1 & NCR1 AND AH5 VERBAL/NUMERICAL

22 CORRELATIONS BETWEEN VCR1 & NCR1 AND WGCTA

23 CORRELATIONS BETWEEN VCR1 & NCR1 AND MAB (N=154)24 CORRELATIONS WITH GRT2 AND JOB PERFORMANCE IN AN INSURANCE SETTING

25 CORRELATIONS WITH GRT2 AND PERFORMANCE CRITERIA IN BANKING

26 CORRELATIONS BETWEEN GRT2 & SERVICE ENGINEER PERFORMANCE

27 CORRELATIONS BETWEEN GRT2 & PRINTER PERFORMANCE CRITERIA (N=70)28 CORRELATIONS BETWEEN GRT2 & PROFICIENCY CRITERIA

29 CORRELATIONS BETWEEN GRT2 & SUCCESSFUL APPLICANT FOR COMPONENT

COURSE

LIST OF TABLES

1THEORETICALOVERVIEWA major reason for using psychomet-

ric tests to aid selection decisions isthat they provide information thatcannot be obtained easily in other

ways. If such tests are not used thenwhat we know about the applicant is

limited to the information that canbe gleaned from an application formor CV, an interview and references. Ifwe wish to gain information about aperson’s specific aptitudes & abilities

and about their personality, atti-tudes and values then we have little

option but to use psychometric tests.In fact, psychometric tests can do

more than simply provide additionalinformation about the applicant.

They can add a degree of reliabilityand validity to the selection proce-dure that it is impossible to achievein any other way. How they do thisis best addressed by examining the

limitations of the informationobtained through interviews, appli-

cation forms and references andexploring how some of these limita-tions can be overcome by using psy-

chometric tests.

1 THE ROLE OF PSYCHOMETRIC TESTS

IN PERSONNEL SELECTION AND

ASSESSMENT

2 THE ORIGINS OF REASONING TESTS

6

THE ROLE OFPSYCHOMETRIC TESTS INPERSONNEL SELECTIONAND ASSESSMENTWhile much useful information canbe gained from the interview, whichclearly has an important role in anyselection procedure, it does nonethe-less suffer from a variety of weak-nesses. Perhaps the most importantof these is that the interview as beenshown to be a very unreliable way tojudge a person’s character. This isbecause it is an unstandardisedassessment procedure. That is to say,each interview will be different fromthe last. This is true even if the inter-viewer is attempting to ask the samequestions and act in the same waywith each applicant. It is preciselythis aspect of the interview that isboth its main strength and its mainweakness. The interview enables usto probe each applicant in depth anddiscover individual strengths andweaknesses. Unfortunately, the inter-views unstandardised, idiosyncraticnature makes it difficult to compareapplicants, as it provides no base lineagainst which to contrast intervie-wees’ differing performances. Inaddition, it is likely that differentinterviewers may come to radicallydifferent conclusions about the sameapplicant. Applicants will responddifferently to different interviewers,quite often saying very differentthings to them. In addition, what anyone applicant might say will beinterpreted quite differently by eachinterviewer. In such cases we have toask which interviewer has formedthe correct impression of the candi-date? This is a question to whichthere is no simple answer.

A further limitation of the inter-view is that it only assesses thecandidate’s behaviour in one setting,and with regard to a small numberof people. How the candidate mightact in different situations and withdifferent people (e.g. when dealingwith people on the shop floor) is notassessed, and cannot be predictedfrom an applicant’s interview perfor-mance. Moreover, the interviewprovides no reliable informationabout the candidate’s aptitudes andabilities. The most we can do is askthe candidate about his strengthsand weaknesses, a procedure thathas obvious limitations. Thus therange, and reliability of the informa-tion that can be gained through aninterview are limited.

There are similar limitations onthe range and usefulness of the infor-mation that can be gained fromapplication forms or CV’s. Whilework experience and qualificationsmay be prerequisites for certainoccupations, in and of themselvesthey do not determine whether aperson is likely to perform well orbadly. Experience and academicachievement is not always a goodpredictor of ability or future success.While such information is importantit may not be sufficient on its own toenable us to confidently choosebetween applicants. Thus aptitudeand ability tests are likely to play asignificant role in the selectionprocess as they provide informationon a person’s potential and not justtheir achievements to date.

7Moreover, application forms tell uslittle about a person’s character. It isoften a candidate’s personality thatwill make the difference between anaverage and an outstanding perfor-mance. This is particularly truewhen candidates have relativelysimilar records of achievement andpast performance. Therefore, person-ality tests can play a major role inassisting selection decisions.

References do provide some usefulinformation but mainly for verifica-tion purposes. While past perfor-mance is undoubtedly a good predic-tor of future performance referencesare often not good predictors of pastperformance. If the name of thereferee is supplied by the applicant,then it is likely that they have chosensomeone they expect to speak highlyof them. They will probably haveavoided supplying the names ofthose who may have a less positiveview of their abilities. Aptitude andability tests, on the other hand, giveus an indication of the applicant’sprobable performance under examconditions. This is likely to be a truereflection of the person’s ability.

What advantages do psychometrictests have over other forms of assess-ment? The first advantage they haveis that they add a degree of reliabilityto the selection procedure thatcannot be achieved without their use.Test results can be representednumerically making it easy both tocompare applicants with each other,and with pre-defined groups (e.g.successful vs. unsuccessful job

incumbents). In the case of personal-ity tests the test addresses the issueof how the person characteristicallybehaves in a wide range of differentsituations and with different people.Thus psychometric tests of personal-ity, aptitude and ability provide arange of information that are noteasily and reliably assessed in otherways. Such information can fillimportant gaps which have not beenassessed by application forms, inter-views and references. It can also raisequestions that can later be directlyaddressed in the interview. It is forthis reason that psychometric testsare being used increasingly inpersonnel selection. Their use adds adegree of breadth to assessment deci-sions which cannot be achieved inany other way.

8THE ORIGINS OFREASONING TESTSThe assessment of intelligence orreasoning ability is perhaps one ofthe oldest areas of research interestin psychology. Gould (1981) hastraced attempts to scientificallymeasure psychological aptitudes andabilities to the work of Galton at theend of the last century. Prior toGalton’s pioneering work, however,interest in this area was aroused byphrenologists’ attempts to assessmental ability by measuring the sizeof people’s heads. Reasoning tests, intheir present form, were first devel-oped by Binet, a French educational-ist who published the first test ofmental ability in 1905.

Binet was concerned with assess-ing the intellectual development ofchildren and to this end invented theconcept of mental age. Questions,assessing academic ability, weregraded in order of difficulty accord-ing to the average age at which chil-dren could successfully completeeach item. From the child’s perfor-mance on such a test it was possibleto derive its mental age. Thisinvolved comparing the performanceof the child with the performance ofthe ‘average child’ from different agegroups. If the child performed at thelevel of the average 10 year old, thenthe child was said to have a mental

age of 10, regardless of its chrono-logical age. From this idea theconcept of the Intelligence Quotient(IQ) was developed by William Stern(1912) who defined it as mental agedivided by chronological age multi-plied by 100. Previous to Stern’spaper chronological age had beensubtracted from mental age toprovide a measure of mental alert-ness. Stern showed that it was moreappropriate to take the ratio of thesetwo constructs, which would providea measure of the child’s intellectualdevelopment relative to other chil-dren. He further proposed that thisratio should be multiplied by 100 forease of interpretation; thus avoidingcumbersome decimals.

Binet’s early tests were subse-quently revised by Terman et al.(1917) to produce the now famousStanford-Binet IQ test. These earlyIQ tests were first used for selectionby the American’s during the firstworld war, when Yerkes (1921)tested 1.75 million soldiers with thearmy alpha and beta tests. Thus bythe end of the war, the assessment ofreasoning ability had firmly estab-lished its place within psychology.

2THE GRADUATE& GENERALREASONING TESTS(GRT1 & GRT2)

bkResearch in the area of intelligencetesting has consistently demon-strated that three aptitude domains:Verbal, Numerical and AbstractReasoning Ability (Heim, 1970).Consequently, the GRT1 and GRT2have been designed to measure justthese three areas of ability. Verbaland Numerical ability assess, as theirrespective names would suggest, theability to use words and numbers ina rational way, correctly identifyinglogical relationships between theseentities and drawing conclusions andinferences from them. Abstract,reasoning assesses the ability to iden-tify logical relationships betweenabstract spatial relationships andgeometric patterns. Many psycholo-gists argue that Abstract Reasoningtests assess the purest form of ‘intel-ligence’. That is to say, these tests arethe least affected by educationalexperience and assess what somemight term ‘innate’ reasoning ability.Namely, the ability to solve abstract,logical problems which require noprior knowledge or educationalexperience.

Research has clearly demonstratedthat in order to accurately assessreasoning ability it is necessary todevelop tests which have beenspecifically designed to measure thatability in the population underconsideration. That is to say, weneed to be sure that the test has beendeveloped for use on the particulargroup being tested, and thus that itis appropriate for that particulargroup. There are two ways in whichthis is important. Firstly, it is impor-tant that the test has been developedin the country in which it is intendedto be used. This ensures that theitems in the test are drawn from acommon, shared cultural experience,giving each candidate an equalopportunity to understand the logicwhich underlies each item. Secondly,it is important that the test isdesigned for the particular abilityrange on which it is to be used. Atest designed for those of averageability will not accurately distinguishbetween people of high ability as alltheir scores will cluster towards thetop end of the scale. Similarly, a testdesigned for people of high abilitywill be of little practical use if givento people of average ability. Not onlywill the test not discriminate betweenapplicants, as all the scores willcluster towards the bottom of thescale, but also as the questions willbe too difficult for most of the appli-cants they are likely to lose motiva-tion, producing artificially lowscores. For this reason two versionsof this reasoning test were developed.One developed for the general popu-lation (the average ability range) andone for the graduate population.

blIn constructing the items in the

GRT1 and GRT2 which measurethese reasoning abilities a number ofguide lines were borne in mind.Firstly, and perhaps most impor-tantly, each item was constructed sothat only a minimal educational levelwas needed in order to be able tocorrectly solve each item. Thus wetried to ensure that each item was ameasure of ‘reasoning ability’, ratherthan being a measure of specificknowledge or experience. Forexample, in the case of the numericalitems the calculations involved insolving each item are relativelysimple, with the difficulty of the itembeing due to the logic which under-lies that question rather than beingdue to the need to use complexmathematical operations to solvethat item. (It should be notedhowever that in the case of the GRT1a higher level of education wasassumed, as the test was designed fora graduate population). Secondly, anumber of different item types (e.g.odd one out, word meanings etc.)were used to measure each aspect ofreasoning ability. This was done inorder to ensure that each sub-scalemeasures a broad aspect of reasoningability (e.g. Verbal ReasoningAbility), rather than measuring avery specific aptitude (e.g. vocabu-lary). In addition, the use of differentitem types ensures that the test ismeasuring different components ofreasoning ability. For example theability to understand analogies,inclusion/exclusion criteria for classmembership etc.

3THE CRITICALREASONING TESTBATTERY (CRTB)

boResearch has clearly demonstratedthat in order to accurately assessreasoning ability it is necessary todevelop tests which have beenspecifically designed to measure thatability in the population underconsideration. That is to say, weneed to be sure that the test has beendeveloped for use on the particulargroup being tested, and thus isappropriate for that particulargroup. There are two ways in whichthis is important. Firstly, it is impor-tant that the test has been developedin the country in which it is intendedto be used. This ensures that theitems in the test are drawn from acommon, shared cultural experience,giving each candidate an equalopportunity to understand the logicwhich underlies each item. Secondly,it is important that the test isdesigned for the particular abilityrange on which it is to be used. Atest designed for those of averageability will not accurately distinguishbetween people of high ability as allthe scores will cluster towards thetop end of the scale. Similarly, a testdesigned for people of high abilitywill be of little use if given to peopleof average ability. Not only will it notdiscriminate between applicants, asall the scores will cluster towards thebottom of the scale, but also as thequestions will be too difficult formost of the applicants they are likelyto be de-motivated, producing artifi-cially low scores. Consequently, the

VCR1 and NCR1 have been devel-oped on data from undergraduates.That is to say, people of aboveaverage intelligence, who are likelyto find themselves in senior manage-ment positions as their career devel-ops.

In constructing the items in theVCR1 and NCR1 a number of guidelines were borne in mind. Firstly,and perhaps most importantly,special care was taken when writingthe items to ensure that in order tocorrectly solve each item it wasnecessary to draw logical conclusionsand inferences from the stempassage/table. This was done toensure that the test was assessingcritical (logical/deductive) reasoningrather than simple verbal/numericalchecking ability. That is to say, theitems assess a person’s ability tothink in a rational, critical way andmake logical inferences from verbaland numerical information, ratherthan simply check for factual errorsand inconsistencies.

In order to achieve this goal forthe Verbal Critical Reasoning(VCR1) test two further points wereborn in mind when constructing thestem passages for the VCR1. Firstly,the passages were kept fairly shortand cumbersome grammaticalconstructions were avoided, so that aperson’s scores on the test would notbe too affected by reading speed;thus providing a purer measure ofcritical reasoning ability. Secondly,

bp

care was taken to make sure that thepassages did not contain any infor-mation which was counter-intuitive,and was thus likely to create confu-sion. To increase the acceptability ofthe test to applicants the themes ofthe stem passages were chosen to berelevant to a wide range of businesssituations. As a consequence of theseconstraints the final stem passageswere similar in many ways to theshort articles found in the financialpages of a daily newspaper.

Finally an extended responseformat was chosen for the VCR1.While many critical reasoning testsonly ask the applicant to make thedistinction between whether thetarget statement is true, false orcannot be inferred from the stempassage the response format for theVCR1 was extended to include thejudgement of whether the targetstatement was probably or definitelytrue/false given the information inthe passage. This was done in orderto decrease the chance of guessing acorrect answer from 33% to 25%.With guessing having substantiallyless impact on a candidate’s finalscore, it was thus possible to decreasethe number of items in the test thatwere needed for it to be reliable.

4THE PSYCHOMETRICPROPERTIES OF THEREASONING TESTSThis chapter will present details con-

cerning the psychometric propertiesof the reasoning Tests. The aim will

be to show that these measures fulfilvarious technical requirements, in

the areas of standardisation, reliabil-ity and validity, which ensure the

psychometric soundness of the test.

1 INTRODUCTION

2 STANDARDISATION

3 RELIABILITY OF THE REASONING

TESTS

4 METHOD EFFECTS

5 VALIDITY

6 THE STRUCTURE OF THE REASONING

TESTS

7 THE CONSTRUCT VALIDITY OF THE

REASONING TEST

8 CRITERION-RELATED VALIDITY

bs INTRODUCTION

Standardisation : NormativeNormative data allows us to comparean individuals score on a standard-ised scale against the typical scoreobtained from a clearly identifiable,homogeneous group of people.

RELIABILITY

The property of a measurementwhich assesses the extent to whichvariation in measurement is due totrue differences between people onthe trait being measure or tomeasurement error.

In order to provide meaningfulinterpretations, the reasoning testswere standardised against a numberof relevant groups. The constituentsamples are fully described in thenext section. Standardisation ensuresthat the measurements obtainedfrom a test can be meaningfullyinterpreted in the context of a rele-vant distribution of scores. Anotherimportant technical requirement fora psychometrically sound test is thatthe measurements obtained fromthat test should be reliable.

Reliability is generally assessedusing two specific measures, onerelated to the stability of scale scoresover time, the other concerned withthe internal consistency, or homo-geneity of the constituent items thatform a scale score.

Reliability : StabilityAlso known as test-retest reliability,an assessment is made of the similar-ity of scores on a particular scaleover two or more test occasions. Theoccasions may be from a few hours,days, months or years apart.

Normally Pearson correlation coeffi-cients are used to quantify the simi-larity between the scale scores overthe two or more occasions.

Stability coefficients provide animportant indicator of a test’s likelyusefulness of measurement. If thesecoefficients are low (< approx. 0.6)then it is suggestive of either that theabilities/behaviours/attitudes beingmeasured are volatile or situationallyspecific, or that over the duration ofthe retest interval, situational eventshave made the content of the scaleirrelevant or obsolete. Of course, theduration of the retest intervalprovides some clue as to which effectmay be causing the unreliability ofmeasurement. However, the secondmeasure of a scales reliability alsoprovides valuable information as towhy a scale may have a low stabilitycoefficient. The most commonmeasure of internal consistency isCronbach’s Alpha. If the items on ascale have high inter-correlationswith each other, and with the totalscale score, then coefficient alphawill be high. Thus a high coefficientalpha indicates that the items on thescale are measuring very much thesame thing, while a low alpha wouldbe suggestive of either scale itemsmeasuring different attributes or thepresence of error.

Reliability : InternalConsistencyAlso known as scale homogeneity, anassessment is made of the ability ofthe items in a scale to measure thesame construct or trait. That is aparameter can be computed that

bt

indexes how well the items in a scalecontribute to the overall measure-ment denoted by the scale score. Ascale is said to be internally consis-tent if all the constituent itemresponses are shown to be positivelyassociated with their scale score.

The fact that a test has high inter-nal consistency & stability coeffi-cients only guarantees that it ismeasuring something consistently. Itprovides no guarantee that the test isactually measuring what it purportsto measure, nor that the test willprove useful in a particular situation.Questions concerning what a testactually measures and its relevancein a particular situation are dealtwith by looking at the tests validity.Reliability is generally investigatedbefore validity as the reliability oftest places an upper limit on testsvalidity. It can be mathematicallydemonstrated that a validity coeffi-cient for a particular test can notexceed that tests reliability coeffi-cient.

VALIDITY

The ability of a scale score to reflectwhat that scale is intended tomeasure. Kline’s (1993) definition is“A test is said to be valid if itmeasures what it claims to measure”.

Validation studies of a test investi-gate the soundness and relevance ofa proposed interpretation of thattest. Two key areas of validation areknown as criterion validity andconstruct validity.

Validity : Criterion ValidityCriterion validity involves translat-ing a score on a particular test into aprediction concerning what could beexpected if another variable wasobserved.

The criterion validity of a test isprovided by demonstrating thatscores on the test relate in somemeaningful way with an externalcriterion. Criterion validity comes intwo forms – predictive & concurrent.Predictive validity assesses whether atest is capable of predicting anagreed criterion which will be avail-able at some future time – e.g. can atest predict the likelihood of someonesuccessfully completing a trainingcourse. Concurrent validity assesseswhether the scores on a test can beused to predict a criterion measurewhich is available at the time of thetest – e.g. can a test predict currentjob performance.

Validity : Construct ValidityConstruct validity assesses whetherthe characteristic which a test isactually measuring is psychologicallymeaningful and consistent with thetests definition.

The construct validity of a test isassessed by demonstrating that thescores from the test are consistentwith those from other major testswhich measure similar constructsand are dissimilar to scores on testswhich measure different constructs.

ck STANDARDISATION

For each of the three reasoningbatteries, information is provided onthe constituent norm samples, ageand gender differences where theyapply. All normative data is availablefrom within the GeneSys systemwhich computes for any given rawscore, the appropriate standardisedscores for the selected referencegroup. In addition the GeneSys™software allows users to establishtheir own in-house norms to allowmore focused comparison withprofiles of specific groups.

GRT2 NORMATIVE DATA

The total norm base of the GRT2 isbased on a general population normas well as a number of morespecialised norm groups. Theseinclude undergraduates, technicalstaff, personnel managers, customerservice staff, management applicantsetc. detailed in Table 1.

GRT2 GENDER AND AGEDIFFERENCES

Gender differences on GRT2 wereexamined by comparing results ofalmost equal numbers of males andfemale respondents matched as faras possible for educational and socio-economic status. Table 2 providesmean scores for males and femaleson each of the GRT2 scales as well asthe t-value for mean score differ-ences.

The results below demonstrategender differences on NR2 with themale mean score just over two rawscore points higher than that of thefemales. This is in line with othernumerical measures. More surprisingperhaps, is that no differences wereobserved on either the Verbal andAbstract measures.

The effect of age on GRT2 scoreswas examined using a sample of1441 respondents on whom age datawas available (see Table 3). Whereasthere is a negative relationship withGRT2 scores and age for all the threemeasures, this tendency is highlysignificant on the Abstract (AR2),suggesting, in line with expectationsthat fluid ability may be more likelyto decline with age than crystallisedability.

cl

General Population

Telesales Applicants

HE College Students

Customer Service Clerks

Technical Staff

Financial Consultants

HR Professionals

Service Engineers

Males N

3177

175

5

28

122

58

30

86

Females N

1236

391

158

87

24

20

38

8

Mean Age

35.10

27.16

16.25

26.46

29.38

32.94

36.31

26.56

Range

16-63

16-55

15-41

19-50

16-58

20-53

22-55

18-58

SD Age

8.87

8.29

3.14

7.19

9.92

9.20

7.94

6.87

Table 1: GRT2 Standardisation Samples

GRT2

VR2

NR2

AR2

MeanFemales

23.36

15.77

16.95

MeanMales

23.22

17.91

17.40

t-value

.3404

-5.424

-1.259

df

1438

1438

1438

p

.734

.000

.208

FemalesN

852

852

852

MalesN

588

588

588

Table 2: Gender differences on GRT2

GRT2

VR2

NR2

AR2

AGE

.11

.10

-.37

Table 3: Pearson correlations between GRT2 and age

cmGRT1 GENDER AND AGEDIFFERENCES

Gender differences on GRT1 wereexamined by comparing results ofalmost equal numbers of males andfemale respondents matched as faras possible for educational and socio-economic status. Table 4 providesmean scores for males and femaleson each of the sub-scales of theGRT1 as well as the t-value for meanscore differences.

Consistent with findings on theGRT2, the only significant scoredifference is registered on theNumerical (NR1) with males regis-tering a higher mean score. Nosignificant differences were observedfor either the Verbal or the Abstractcomponents of the GRT1.

The effect of age on GRT1 scoreswas examined using a sample of 499respondents on whom age data wasavailable (see Table 5). Whereas theVerbal and Numerical were found tobe unrelated to age, the Abstractshowed a significant negative rela-tionship, consistent with expecta-tions. The correlations are lowerthan those obtained with the GRT2,although this may be explained bythe sample which in this case is morerestricted in the range of observedscores.

CRTB GENDER AND AGEDIFFERENCES

Gender differences on CRTB wereexamined by comparing results ofmales and female respondentsmatched as far as possible for educa-tional and socio-economic status.Table 6 opposite provides meanscores for males and females on boththe verbal and numerical sub-scaleof the CRTB scales as well as the t-value for mean score differences.

While female respondents registermarginally higher verbal reasoningscores (VCR1) than males, this is notstatistically significant. A significantgender difference was observed onthe Numerical (NCR1), with malesregistering over four raw score pointshigher mean score. This is consistentwith other measures of numericalability.

The effect of age on CRTB scoreswas examined using a sample of 359respondents on whom age data wasavailable (see Tabel 7). Unlike theGeneral and Graduate Reasoningtests, the correlations with age arepositive for the Critical Reasoning,although only marginal and non-significant for the Verbal. Thissuggests that the Numerical at least,may be measuring more of anacquired ability, which is more posi-tively influenced by experience thanother more classic measures ofnumerical ability.

cn

GRT1

VR1

NR1

AR1

MeanFemales

16.73

12.98

14.11

MeanMales

16.61

14.91

14.27

t-value

.260

-4.255

-.383

df

497

497

497

p

.795

.000

.702

FemalesN

251

251

251

Males N

248

248

248

Table 4: Gender Differences on GRT1

CRTB

VCR1

NCR1

F

18.34

10.19

M

17.78

14.40

t-value

.501

-3.578

df

134

134

p

.6178

.0004

F

47

47

M

89

89

Table 6: Gender differences on CRTB

GRT1

VR1

NR1

AR1

AGE

-.00

-.05

-.27

Table 5: Relationship between Age and GRT1

CRTB

VRC1

NRC1

AGE

.03

.18

Table 7: Relationship between Age and CRTB

co

If a reasoning test is to be used forselection and assessment purposesthe test needs to measure each of theaptitude or ability dimensions it isattempting to measure reliably, forthe given population (e.g. graduateentrants, senior managers etc.). Thatis to say, the test needs to be consis-tently measuring each ability so thatif the test were to be used repeatedlyon the same candidate it wouldproduce similar results.

It is generally recognised thatreasoning tests are more reliablethan personality tests and for thisreason high standards of reliabilityare usually expected from such tests.While many personality tests areconsidered to have acceptable levelsof reliability if they have reliabilitycoefficients in excess of .7, reasoningtests should have reliability coeffi-cients in excess of .8.

GRT2 INTERNAL CONSISTENCY

Table 8 presents alpha coefficientsfor the three sub-scales of the GRT2(n=135). Each of these reliabilitycoefficients is substantially greaterthan .8, clearly demonstrating thatgeneral population version of theGRT is highly reliable.

GRT1 INTERNAL CONSISTENCY

Table 9 presents alpha coefficientsfor the three sub-scales of the GRT1(n=109). Each of these reliabilitycoefficients is greater than .8, clearlydemonstrating that the graduateversion of the GRT meets acceptablelevels of internal consistency.

GRT1 TEST-RETEST

A sample of 70 undergraduatestudents completed the GRT1 on twoseparate occasions with a four weekinterval. Table 10 provides uncor-rected correlations for each measure.

Although the Abstract falls some-what below the ideal, the test-retestcorrelations are generally of a highand acceptable level which, inconjunction with the internal consis-tency data would demonstrate thatthe GRT1 is a reliable measure ofgeneral reasoning ability.

Table 11 presents alpha coeffi-cients for the two sub-scales of theCritical Reasoning Test. Each ofthese reliability coefficients is .8 orgreater, clearly demonstrating thatCritical Reasoning Tests reachacceptable levels of reliability.

RELIABILITY OF THEREASONING TESTS

cp

GRT2

Verbal (VR2)

Numerical (NR2)

Abstract (AR2)

r

.83

.84

.83

Table 8: Coefficient Alpha for GRT2Sub-scales (n = 135)

GRT1

Verbal (VR1)

Numerical (NR1)

Abstract (AR1)

r

.82

.85

.84

Table 9: Coefficient Alpha for GRT1Sub-scales (n = 109)

CRTB

Verbal Critical Reasoning (VCR1)

Numerical Critical Reasoning (NCR1)

r

.80

.82

Table 11: Coefficient Alpha for CRTB Sub-scales (N=134)

GRT1

VR1

NR1

AR1

VR1

.79

.35

.22

NR1

.31

.78

.38

AR1

.09

.39

.74

Table 10: Test-retest reliability esti-mates for GRT1 (N=70)

cq METHOD EFFECTS

One important aspect of the changefrom paper and pencil to computer-based administration, concerns theeffects the change in test formatmight have on the available norma-tive data for a test. In other wordsdoes the translation of a test tocomputer format alter the nature ofthe test itself? It is possible that thescore range of a test administered inquestion booklet form may be differ-ent from the range when the test isadministered via a computer. Giventhat most of the normative dataavailable for mainstream psychomet-ric tests was collected from paperand pencil test administration thequestion is far from academic. Forinstance, many organisations willadminister computer-based tests attheir head office location but will usepaper and pencil format when theirtesters visit remote locations. Ifcomparison is to be made across agroup in which some individualsreceived paper and pencil testing andsome computer testing then theimportance of the question ofnormative comparability can not beoverstated.

Very little research attention hasbeen paid to this topic, which issurprising given the possible impactthat format differences would have.Roid (1986) has indicated that whatlittle evidence is available concerningpaper and pencil v. computerformats suggests that computeradministration of most tests does notchange the score range enough toaffect the normative basis of the test.While this is somewhat reassuringthe number of studies looked at by

Roid was small and consistedentirely of American investigations.

As publishers of a wide range oftests which can be administered bothby paper and pencil and computer,Psytech International recentlyconducted a study looking at theeffect of administration format onreasoning test performance. The Testchosen was the Graduate ReasoningTest, a graduate level test comprisingthree sub-tests – verbal, numericaland abstract. This test was chosen asbeing representative of many main-stream reasoning tests. The GraduateReasoning Test also gives an oppor-tunity to test whether the representa-tional format of a test is important.It could be the case that computerpresentation of graphical material,such as is found in mechanical andspatial reasoning tests, might lead toperformance differences while alpha-numeric material does not. With theGRT1 the abstract subtest uses agraphical format while the verbaland numerical sub-scales use alpha-numeric formats.

cr

THE STUDY

A group of 80 university undergrad-uates took part in this investigation.The students were divided into fourgroups. Each group completed theGraduate Reasoning Test twice withan interval of two weeks betweensuccessive administrations. Two ofthe groups completed the sameformat of the test on each occasion –i.e. paper/paper orcomputer/computer. The other twogroups experienced both formats ofthe test, each group in a differentorder – i.e. paper/computer orcomputer/paper.

RESULTS

An independent t-test was used totest whether any differences existedbetween the mean scores on firstadministration for those studentswho completed the paper version ofthe GRT1 and those receiving theGeneSys Administered version. Ascan be seen from Table 12 a signifi-cant difference was found for boththe Numerical and Abstract sub-scales in that students who hadreceived the computer administeredversion of the test tended to scoresignificantly higher than those whohad received the paper version first.Table 12 provides similar data forthe two test formats on secondadministration. It can be seen fromthis table that no significant differ-ences existed between the differentformats for second administration.

CONCLUSION

The results of this study show that asfar as the Graduate Reasoning Test,at least, is concerned the format inwhich the test is administered doesaffect, to some extent, the scoresobtained. No differences between thegroup means were detected for eitherof the three sub-tests of the GraduateReasoning Test. It was also the casethat no difference was foundbetween the graphical representationof the abstract sub-test and thealpha-numerical representation ofthe verbal and numerical sub-tests.Furthermore no interaction effectswere observed which indicates thatthe well established phenomenon of‘practice effects’ does not differ withthe nature of the test medium.

These results provide some confi-dence that the changeover frompaper and pencil to computer-basedtesting will not require the re-stan-dardisation of tests. It would seemfrom this study that administrationof ability tests by either computer orpaper and pencil will produce similarperformance levels. Thus it isperfectly acceptable to compare indi-viduals who were tested using differ-ent formats.

GRT1

VR1

NR1

AR1

Mean

PC

19.09

14.53

15.58

Mean

Paper

18.48

13.86

14.79

t-value

.876

.927

1.117

df

158

158

158

p

.382

.355

.266

N-PC

80

80

80

N-Paper

80

80

80

SD-PC

4.26

3.97

3.81

SD-

Paper

4.58

5.01

5.03

F-ratio

variance

1.15

1.59

1.74

P

variance

.532

.0419

.0146

Table 12: Differences between Computer vs. Conventional Methods of Test Administration

cs VALIDITY

Whereas reliability assess the degreeof measurement error of a reasoningtest, that is to say the extent to whichthe test is consistently measuring oneunderling ability or aptitude, validityaddresses the question of whether ornot the scale is measuring the char-acteristic it was developed tomeasure. This is clearly of keyimportance when using a reasoningtest for assessment and selectionpurposes. In order for the test to be auseful aid to selection we need to

know that the results are reliable andthat the test is measuring the apti-tude it is supposed to be measuring.Thus after we have examined a test’sreliability we need to address theissue of validity. We traditionallyexamine the reliability of a testbefore we explore its validity as reli-ability sets the lower bound of ascale’s validity. That is to say a testcannot be more valid than it is reli-able.

GRT1 sub-scale

Verbal (VR1)

Numerical (NR1)

Abstract (AR1)

VR1

_

NR1

.42

_

AR1

.30

.55

_

Table 13: Product-moment Correlationsbetween the GRT1 Sub-scales (n=499)

GRT2 sub-scale

Verbal (VR2)

Numerical (NR2)

Abstract (AR2)

VR2

_

NR2

.60

_

AR2

.56

.65

_

Table 14: Product-moment Correlationsbetween the GRT2 Sub-scales (n = 1441)

ctTHE STRUCTURE OF THEREASONING TESTS

Specifically we are concerned thatthe test’s sub-scales are correlatedwith each other in a meaningful way.For example, we would expect thedifferent sub-scales of a reasoningtest to be moderately correlated aseach will be measuring a differentfacet of general reasoning ability.Thus if such sub-scales are not corre-lated with each other we mightwonder whether each is a goodmeasure of reasoning ability.Moreover, we would expect differentfacets of verbal reasoning ability(e.g. vocabulary, similarities etc.) tobe more highly correlated with eachother than they are with a measureof numerical reasoning ability.Consequently, the first way in whichwe might assess the validity of areasoning test is by exploring therelationship between the test’s sub-scales.

THE GRADUATE REASONINGTESTS (GRT1)

Table 13, which presents PearsonProduct-moment correlationsbetween the three sub-scales of theGRT1 demonstrates two things.Firstly, the relatively strong correla-tions between each of the sub-scalesindicate that each is measuring onefacet of an underlying ability. This isclearly consistent with the design ofthis test, where each sub-scale wasintended to assess a different facet ofreasoning ability or mental alertness.Secondly, the fact that each sub-scaleaccounts for less than 30% (r < .55)of the variance in the other sub-scales indicates that the Verbal,Numerical and Abstract Reasoningsub-scales of the GRT1 are measur-ing different facets of reasoningability, as they were designed to.Moreover, this is what we would infact predict from research in the areaintelligence testing (Heim, 1970).

THE GENERAL REASONINGTESTS (GRT2)

Table 14, which presents PearsonProduct-moment correlationsbetween the three sub-scales of theGRT2 demonstrates two things.Firstly, the relatively strong correla-tions between each of the sub-scalesindicate that each is measuring oneunderlying characteristic, which inthis case we might assume to bereasoning ability or mental alertness.Thus these relatively strong correla-tions between the sub-scales areconsistent with our intention toconstruct a test which measuresgeneral reasoning ability. Secondly,the fact that each sub-scale accountsfor less than 45% (r < .65) of thevariance in the other sub-scales indi-cates that the Verbal, Numerical andAbstract Reasoning sub-scales of theGRT2 are still measuring distinctaspects of reasoning ability.

THE CRITICAL REASONINGTESTS (CRTB)

Table 15, which presents the PearsonProduct moment correlation betweenthe two sub-tests of the CRTB,demonstrates that while the Verbaland Numerical sub-tests are margin-ally correlated, they neverthelessmeasuring quite distinct abilities,sharing only 10% of common vari-ance.

CRTB

Verbal

Numerical

Verbal

1

.352

Numerical

.352

1

Table 15: Product-moment Correlationsbetween CRTB Verbal & Numerical (n=352)

dkTHE CONSTRUCT VALIDITYOF THE REASONING TESTS

As an evaluation of construct valid-ity, the Psytech Reasoning Tests wereadministered with other widely usedmeasures of similar constructs.

In the case of the General andGraduate reasoning tests, the AHseries was considered to be a suitableexternal measure. The AH Series oftests is one of the most widelyrespected range of reasoning testswhich have been developed on aU.K. population. Within this seriesthere are tests which have beenspecifically designed for use on boththe general population (AH2/ AH3/AH4) and the graduate population(AH5/AH6). Developed by AliceHeim (1968, 1974) of CambridgeUniversity the AH series has becomesomething of a benchmark againstwhich to compare the performanceof other reasoning tests.

As the original Critical ThinkingAppraisal, The Watson-Glaser (W-GCTA) (Watson & Glaser (1991) hasset the standard in the assessment ofabilities that are of relevance inmanagement decision-making.

Thus we chose to explore theconstruct validity of the GRT1 andthe GRT2 by comparing their perfor-mance against that of the AH3 andAH5 respectively, while the constructvalidity of the CRTB is examined bycomparing its performance againstboth the AH5 and the Watson-GlaserCritical Thinking Appraisal.

THE RELATIONSHIP BETWEENTHE GRT1 AND AH5

Table 16 presents product-momentcorrelations between the sub-scalesand the total scale scores of the AH5and the GRT1. The AH5, which hasbeen developed for use on a graduatepopulation, has a Verbal/Numericalsub-scale which combines verbal andnumerical items and an Abstract, orDiagrammatic, reasoning sub-scale.These along with the total scale scoreon the AH5 (the sum of the sub-scales) were correlated with theGRT1 sub-scale scores and the totalscale score. The correlations with thetotal scale scores were includedwithin this table, even though thetotal scale scores are simple compos-ites of the sub-scale scores, as theyprovide a measure of general (g),rather than specific (e.g. verbal,numerical etc.) mental aptitudes.

Table 16 provides clear supportfor the concurrent validity of theGRT1 against the AH5. The correla-tions between each of the GRT1 sub-scales and their comparable AH5sub-scales are high, indicating thatthey are measuring similarconstructs. In addition, this tableprovides some evidence in support ofthe discriminant validity of thesesub-scales. That is to say, each of theGRT1 sub-scales is more highlycorrelated with its comparable sub-scale on the AH5 than it is with theAH5 sub-scale measuring a differentspecific mental ability. For example,while the VR1 has a correlation of.69 with the Verbal/Numerical sub-

dl

scale of the AH5, its correlation withthe Abstract sub-scale is only .35. Inaddition, the extremely high correla-tion (r=.84) between the total scalescores of the AH5 and the GRT1 indi-cates that, as a whole, this battery oftests is a good measure of generalreasoning ability (g).

THE RELATIONSHIP BETWEENTHE GRT2 AND AH3

Table 17 presents Pearson Product-moment correlations (n=81) betweeneach of the GRT2 sub-scales with eachof the sub-scale scores of the AH3 andthe total scale score. In addition to thehigh correlations between these sub-scales it is also worth noting theextremely high correlation (r=.82)between the total scale scores on thesetwo tests. These results clearly demon-strate that the GRT2 is measuring thetrait of general reasoning ability whichis assessed by the AH3. We shouldhowever note that the correlationsbetween the sub-scales provide noclear support for the discriminantvalidity of the GRT2. That is to say,the correlations between each of theGRT2 sub-scales and their respectiveAH3 sub-scales (e.g. the VR2 with theAH3 Verbal) are not significantlyhigher than are the correlations acrosssub-scales (e.g. the VR2 with the AH3Numerical and Verbal).

SUB-SCALE

AH5 Verbal/Numerical

AH5 Abstract

AH5 Total

VR1

.69

.51

.69

NR1

.70

.67

.79

AR1

.35

.72

.65

TOTAL

.70

.74

.84

Table 16: Product-moment correlations between the GRT1 andAH5 Sub-scales

SUB-SCALE

AH5 Verbal

AH3 Numerical

AH5 Perceptual

AH5 Total

VR2

.63

.58

.54

.70

NR2

.63

.76

.55

.78

AR2

.61

.76

.56

.78

GRTTotal

.73

.70

.76

.82

Table 17: Product-moment Correlations between theGRT2 and AH3 Sub-scales

dm

RELATIONSHIP WITH GRT2AND OTHER MEASURES

As a part of a number of data-collec-tion exercises, the GRT2 was appliedwith a number of alternativemeasures, namely the technical TestBattery (TTB2) and Clerical TestBattery (CTB2).

Technical Test Battery (TTB2)A sample of 94 trainee Mechanicalapprentices completed both theGRT2 and the Technical Test Batteryas part of a validation exercise. TheGRT2 sub-scales register modestcorrelations with the componentsmeasures of the Technical TestBattery, although this is no morethan would be expected from differ-ent aptitude measures with noneexceeding .50 (see Table 18).

Clerical Test Battery (CTB2)A sample of 54 clerical staff workingfor a major bank completed theVerbal reasoning Test (VR2) as partof an assessment of Clerical apti-tudes which included components ofthe Clerical Test Battery (CTB2).

The strongest observed correlationwas not with the Spelling measure(SP2) as expected but with OfficeArithmetic (NA2). Examination ofNA2 does reveal a fairly high verbalproblem-solving element, which mayexplain this. The Clerical CheckingTest (CC2) only registered a modestcorrelation which is as expected of ameasure which relies only to alimited extent on general ability (seeTable 19).

THE RELATIONSHIP BETWEENTHE CRTB AND AH5

In Tables 20 and 21 we present twosets of data supporting the concur-rent validity of the VCR1 and NCR1.The first data set was collected trialversions of these two tests whichcontained each of the stempassages/tables which appear in thefinal tests along with approximately80% of the final items. This datawas originally collected as part of thetest construction process in order tocheck that the trial items we hadconstructed were measuring reason-ing ability, and not some otherconstruct (e.g. reading ability,numerical checking etc.). This isparticularly important whenconstructing critical reasoning testsas it is easy to construct items whichare better measures of checkingability than they are of reasoningability. That is to say, items whichsimply require carefully scanningand memorising the text in order tosuccessfully complete them, ratherthan having to correctly drawslogical inferences from the text. Ascan be seen from the table, our trialitems clearly appear to be measuringability, as scores on both the VCR1and NCR1 are strongly correlatedwith the AH5.

Table 21 presents the correlationsbetween the Verbal/Numerical sub-scale of the AH5 and the finalversion of the two critical reasoningtests. This data therefore providesevidence in support of the fact thatthe final versions of these two tests

dn

measure reasoning ability rather thansome other construct (i.e. verbal ornumerical checking ability). As wasnoted above, when developing criticalreasoning tests it is particularly impor-tant to demonstrate that the tests aremeasuring reasoning ability, and notchecking ability.

The above data clearly demon-strates, as does the previous data set,that both the VCR1 & NCR1 aremeasuring reasoning ability. The sizeof these correlations indicate that theVerbal/Numerical sub-scale of theAH5 and the two critical reasoningtests share no more than 35% ofcommon variance, clearly demonstrat-ing that these tests are measuringdifferent, but related, constructs. Thisis what we would predict, given thatthe VCR1 & NCR1 were developed tomeasure critical reasoning, rather thanbe ‘pure’ measures of mental ability, orintelligence. Given the nature of criti-cal reasoning, we would expect acandidate’s scores on these tests notonly to reflect general reasoning abilityor intelligence (g), but also to reflectverbal and numerical comprehension,reading ability, reading speed andnumerical ability and precision.

Technical Test Battery

MRT2

SRT2

VAC2

VR2

.45

.35

.34

NR2

.45

.47

.40

AR2

.38

.46

.40

Table 18: Pearson Correlations of GRT2 withTechnical test Battery

CTB2 Sub-scales

Office Arithmetic (NA2)

Clerical Checking (CC2)

Spelling (SP2)

VR2

.51

.37

.34

Table 19: Pearson Correlations ofGRT2 with Clerical Test Battery

CRTB

VRC1

NRC1

AH5

.51

.52

Table 20: Product-moment Correlationsbetween the Experimental Versions of theVCR1 & NCR1 and the Verbal NumericalSub-scale of the AH5

CRTB

VRC1

NRC1

AH5

.60

.51

Table 21: Product-moment Correlationsbetween VCR1 & NCR1 and AH5Verbal/Numerical

do

RELATIONSHIP BETWEEN THECRTB AND WATSON-GLASER CRITICAL THINKING APPRAISAL

Table 22 details the relationshipbetween the CRTB and the Watson-Glaser Critical Thinking Appraisal.The relationship between the totalscore on CRTB and W-GCTA ismoderate, although this may be dueto its absence of numerical content.However, unexpectedly, the CRTBVerbal sub-scale does not appear tohave a higher correlation with the W-GCTA than the numerical. Insummary, while CRTB does notappear to be measuring exactly thesame construct as the W-GCTA, thedomains do overlap to such anextent that this provides someevidence that the CRTB is a measureof critical thinking.

THE RELATIONSHIP BETWEENTHE CRTB AND THE MULTI-DIMENSIONAL APTITUDE BATTERY

The CRTB was correlated withJackson’s MAB to assess the relativepositioning of the CRTB within abroad range of abilities. Table 23details the relationship between eachMAB sub-scale and the Verbal andNumerical components of the CRTBas well as the CRTB total score. TheCRTB total correlates .60 with theMAB total and is equally related toboth the MAB Verbal andPerformance scales (.57 and .52).More specifically at the sub-scalelevel, the CRTB total, relates signifi-cantly to three of each of the Verbaland Performance sub-scales. It ismore strongly related to theInformation, Arithmetic and ObjectPerception sub-scales (.60, .58, .58).This it would appear that the CRTBtotal measures a composite of bothVerbal and Performance abilities.

When the CRTB is divided into itssub-scales, the Verbal does notappear to be strongly related to theMAB Total but is related to the MABverbal scales in particular,Information and Vocabulary.

As expected, the Numericalsubscale relates more closely with theMAB Performance scale (.48) andalso correlates significantly withMAB Arithmetic and Spatial as itdoes with the Verbal which can beexpected by the higher verbalcontent in VCR1 than more classicmeasures of verbal ability.

dp

Verbal

Numerical

CRTB Total

WGCTA

.38

.38

.57

Table 22: Product-moment Correlationsbetween VCR1 & NCR1 and WGCTA

MAB Total

MAB Performance

MAB Verbal

Information

Comprehension

Arithmetic

Similarities

Vocabulary

Digit Symbol

Picture Completion

Spatial

Picture Arrangement

Object Assembly

Verbal

.24

.04

.43

.29

.25

.24

.22

.32

.09

.14

.19

.15

.09

Numerical

.48

.48

.44

.32

.44

.45

.33

.27

.37

.38

.50

.34

.23

Total

.60

.52

.57

.60

.52

.58

.39

.40

.39

.44

.58

.42

.44

Table 23: Product-moment Correlations between VCR1 &NCR1 and MAB (n=154)

dqCRITERION-RELATEDVALIDITY

In this section, we provide details ofnumber of studies in which thereasoning tests have been used aspart of a pilot study on a sample ofjob incumbents on whom perfor-mance data was available.

INSURANCE SALES

A sample of 86 Telephone SalesPersonnel with a leading MotorInsurance group completed theGRT2 as part of a validation study.The results of the GRT2 were corre-lated with a number of performancecriteria as detailed in Table 24.

The pattern of correlationssuggests that while there is a rela-tionship between GRT2 and perfor-mance measures, this is not alwaysin the anticipated direction. In factsome notable negative correlationswere observed, indicating that forsome performance criteria, higherreasoning ability may be less desir-able. However, the strongest correla-tions were found between GRT2 sub-scales and ‘Ins_Nce’ and the VR2with overall sales. The consistentnegative correlations with ‘Sales PH’would require further examination.

BANKING

A sample of 118 retail bankerscompleted the GRT2 and a personal-ity measure OPP as part of a concur-rent validation exercise. Participantswere rated on a range of competen-cies which focused primarily onpersonal qualities as opposed to abil-ities. As expected, GRT2 failed torelate strongly to the overall compe-tency rating which based on acomposite of ratings covering suchdiverse areas as Orderliness,Planning, Organising, Teamwork etc.

GRT2 did correlate with thoseratings which were associated withskill areas namely, numerical andsoftware related work (see Table 25).

SERVICE ENGINEERS

A leading International Crane &heavy lifting equipment servicingcompany tested a sample of 46service engineers on the GRT2battery and OPP. Their overallperformance was rated by supervi-sors on a behaviourally anchored fivepoint scale. From Table 26, theVerbal (VR2) is found to be stronglyrelated to rated performance (r=.46)and the Abstract (AR2) relatingmoderately with the same.

PRINTERS

A major local newspaper group withthe largest number of local titles inthe United Kingdom sought toexamine whether tests could predictthe job performance of experiencedprinters. A sample of 70 completedthe GRT2 battery as well as anumber of other measures includingthe OPP (Occupational PersonalityProfile) and TTB (Technical TestBattery). Each of the group wereassessed on a number of perfor-mance criteria by supervisors. Inaddition, test data were correlatedwith the results of a job sample printtest which was administered at selec-tion stage. Table 27 details theresults of this study.

Some noteworthy correlationswere registered with GRT2 andperformance measures. Firstly,overall performance is highly corre-lated with the Abstract (AR2) butalso moderately with the Verbal andNumerical sub-scales. The job

dr

sample criterion measure generallyregisters higher correlations with eachof the GRT2 sub-scales, reaching .41with the Abstract (AR2). Perhaps moresurprising, the Abstract correlates .56with a supervisor’s rating of Initiative,consistent with the behaviouraldescriptions used for this performancerating, which point to not simplytaking initiative but being able tosuccessfully resolve problem situations.

The single totally objectivelyderived performance measure wastime-keeping, which was based on anelectronic time and attendance record-ing system. Only one significant corre-lation was observed with Abstract(AR2) although this was negative,suggesting that those with higherability tended to have a poorer timeand attendance record.

Sales PH

Conversions%

Ins_Nce

Sales

Verbal

-.26

-.20

.32

.27

Numerical

-.27

-.21

.25

.14

Abstract

-.27

-.29

.33

.13

Table 24: Pearson Product Moment Correlations withGRT2 and job performance in an Insurance Setting

Sales PH Sales per hourConversions % Percent of Conversions from other policiesIns_NCE Composite Training Outcome MeasureSales Total Value of Policies sold

GRT2

VR2

NR2

AR2

PERF_ARI

.14

.29

.31

PERF_SOF

.03

.32

.28

Competency

-.02

.12

.01

Table 25: Pearson Product Moment Correlations withGRT2 and Performance criteria in Banking (N=118)

GRT2

Measure

Verbal

Numerical

Abstract

Overall

Performance

.46

-.24

.28

Table 26: Pearson ProductMoment correlations betweenGRT2 & Service EngineerPerformance

Criterion

Overall Performance

Performance Job Sample

Initiative

Time Keeping

Verbal

.26

.33

.40

_

Numerical

.28

.30

.44

_

Abstract

.36

.41

.56

.32

Table 27: Correlations Between GRT2 & Printer PerformanceCriteria (N=70)

ds

FINANCIAL SALESCONSULTANTS

A sample of 100 trainee FinancialConsultants from a major financialservices group completed the GRT2as part of a validation study. Table28 details the correlations betweenGRT2 and end of year examinations.

The results indicate that theNumerical (NR2) appears to be thebest predictor of examination perfor-mance with correlations (up to .46),although the Abstract (AR2) alsoregisters some highly notable corre-lations (up to .44). Only the Verbal(VR2) fails to relate to any of theexamination results which is perhapssomewhat unexpected.

TRAINING APPLICANTS FORCAR COMPONENT TRAINING COURSE

A large training company used theGeneral Reasoning Test to investi-gate the ability profiles of success-ful/non-successful applicants fortraining on a car components assem-bly task. The criterion of successused was only in part determined bytraining outcomes, with other factorsalso played a part, such as perceivedattitude, temperament etc. A sampleof 150 applicants was used for thestudy.

Both the Verbal (VR2) andAbstract (AR2) register moderatecorrelations with success on theprogramme. The numerical fails torelate with this criterion (see Table29).

GRT2 Sub-scale

Verbal

Numerical

Abstract

Criterion

Protection Clusters Result

Pension exam Results

Seller Induction Exam Results

Aggregate Result

Financial Planning Certificate Result

VR2

.10

.04.

.18

.11

.13

NR2

.31

.40

.26

.46

.44

AR2

.35

.32

.32

.44

.42

Table 28: Correlations between GRT2 & Proficiency Criteria

Success

.27

.16

.30

Table 29: Correlations betweenOPP & Successful Applicant forComponent Course

5ADMINISTRATIONINSTRUCTIONS● CRTB Administration Instructions

● GRT1 Administration Instructions

● GRT2 Administration Instructions

ekBEFORE STARTING THE QUESTIONNAIRE

Put candidates at their ease by giving information about yourself, thepurpose of the questionnaire, the timetable for the day, if this is part of awider assessment programme, and how the results will be used and who willhave access to them. Ensure that you and other administrators have switchedoff mobile phones etc.

The instructions below should be read out verbatim and the same scriptshould be followed each time the CRTB is administered to one or morecandidates. Instructions for the administrator are printed in ordinary type.Instructions designed to be read aloud to candidates are a larger typefacewith speech marks.

If this is the first or only questionnaire being administered give an introduc-tion as per or similar to the following example (prepare an amendment if notadministering all tests in this battery):

‘From now on, please do not talk among yourselves, butask me if anything is not clear. Please ensure that anymobile telephones, pagers or other potential distractions areswitched off completely. We shall be doing two tests: theVerbal Critical Reasoning Test, which takes 20 minutes andthe Numerical Critical Reasoning Test, which takes 30minutes. During the test I shall be checking to make surethat you are not making any accidental mistakes whenfilling in the answer sheet. I will not be checking yourresponses to see if you are answering correctly or not.’

WARNING: It is most important that answer sheets do not go astray. Theyshould be counted out at the beginning of the test and counted in again atthe end.

Continue by using the instructions EXACTLY as given. Say:

DISTRIBUTE THE ANSWER SHEETS

Then ask:

‘Has everyone got two sharp pencils, an eraser, some roughpaper and an answer sheet. Please note that the answerboxes are in columns (indicate) and remember, do not writeon the booklets.’

Rectify any omissions, then say:

CRTB ADMINISTRATIONINSTRUCTIONS

el‘Print your surname, first name and title clearly on the lineprovided, followed by your age and sex. Please inserttoday’s date which is [ ] on the “Comments” line’

Walk around the room to check that the instructions are being followed.

WARNING: It is vitally important that test booklets do not go astray. Theyshould be counted out at the beginning of the session and counted in again atthe end.

DISTRIBUTE THE BOOKLETS WITH THE INSTRUCTION:

‘Please do not open the booklet until instructed.’

Remembering to read slowly and clearly, go to the front of the group and say:

‘Please open the booklet at Page 2 and follow the instruc-tions for this test as I read them aloud.’

Pause to allow booklets to be opened.

‘In this test you have to draw inferences from shortpassages of text. You will be presented with a passage(there are 6 pieces of text in all) followed by a number ofstatements. Your task is to decide whether or not eachstatement can be inferred from the passage.

For each statement you will be asked to decide which of thefive response categories on Page 3 correctly describeswhether the statement can be inferred from the passage.

Mark your answer by filling in the appropriate box on youranswer sheet that corresponds to your choice. On Page 3 ofthis question booklet there are some example questions.Once you have fully read the instructions you will have achance to complete the example questions in order to makesure that you understand the test.

Please read the instructions and categories.’

Pause while the candidates read the instructions, then say:

‘Please attempt the example questions now.’

While the candidates are doing the examples, walk around the room to checkthat everyone is clear about how to fill in the answer sheet. Make sure thatno-one is looking at the actual test items during the example session. Whenall have finished (allow a maximum of two and a half minutes) give theanswers as follows:

em‘The correct response to Example 1 is Probably False.It is not explicitly stated within the text that voters areconcerned about the economy.

The correct response to Example 2 is Definitely TrueIt is explicitly stated that economic policies are often deter-mined by short term electoral demands.

The correct response to Example 3 is Definitely FalseIt explicitly states within the text that election results canbe predicted from economic indicators.’

Check for understanding, then say:

‘Time is short so when you begin the timed test work asquickly and as accurately as you can.

If you want to change an answer, fully erase your firstchoice, and fill in your new choice of answer.

There are 6 passages of text and a total of 36 questions.You have 20 minutes in which to answer the questions.

If you reach the “End of Test” before time is called youmay review your answers if you wish.

If you have any questions please ask now, as I will not beable to answer questions once the test has started.’

Then say very clearly:

‘Is everyone clear about how to do this test?’

Deal with any questions appropriately, then, starting stop-watch or setting acount-down timer on the word BEGIN say:

‘Please turn over the page and begin.’

Answer only questions relating to procedure at this stage, but enter in theAdministrator’s Test Record any other problems which occur. Walk aroundthe room at appropriate intervals to check for potential problems.

At the end of the 20 minutes, say:

‘Please now turn to Page 10 which is a blank page.’

NB: If this is the final test to be used in this battery, instead of the aboveline, please turn to the instructions in the Ending Test Session box on page45 of this document

enYou should intervene if candidates continue after this point.

Then say:

‘We are now ready to start the next test. Has everybody stillgot two sharpened pencils, an eraser and some unusedrough paper?’

If not, rectify, then say:

‘The next test follows on the same answer sheet, pleaselocate the section now.’

Check for understanding, then, remembering to read slowly and clearly, go tothe front of the group and say:

‘Please turn to Page 12 of the booklet and follow theinstructions for this test as I read them aloud.’ (Pause toallow page to be found).

In this test you have to draw inferences from numericalinformation which is presented in tabular form.

You will be presented with a numerical table and asked anumber of questions about this information. You will thenhave to select the correct answer to each question from oneof six possible choices. One and only one answer is correctin each case.

Mark your answer by filling in the appropriate box on youranswer sheet that corresponds to your choice.

You now have a chance to complete the example questionson Pages 13 in order to make sure that you understand thetest.

Please attempt the example questions now, using the roughpaper if necessary.’

eoWhile the candidates are doing the examples, walk around the room to checkthat everyone is clear about how to fill in the answer sheet. Make sure thatno-one is looking at the actual test items during the example session. Whenall have finished (allow a maximum of three minutes) give the answers asfollows:

‘Please follow the answers and explanations carefully.

The correct answer to Example 1 is Design (answer no. 5).Amongst women, design was consistently chosen by thelowest percentage as the most important feature of a car.

The correct answer to Example 2 is Economy (answer no.2).Of all the features of a car, economy is the only feature forwhich the ratings increase according to age category.

The correct answer to Example 3 is 10.4 (answer no.5).Of males below 30, 5% identified safety and 52% identifiedperformance as the most important feature of a car. 5 over52 is 10.4.

Please do not turn over the page yet’

Then say:

‘Time is short so when you begin the timed test work asquickly and as accurately as you can.


There are 5 tables of information and a total of 25 ques-tions. You have 30 minutes in which to answer the ques-tions.

If you reach the “End of Test” before time is called youmay review your answers if you wish, but do not go back tothe Verbal Reasoning Test.

If you have any questions please ask now, as you will not beable to ask questions once the test has started.’



ep




At the end of the 30 minutes:

ENDING THE TEST SESSION

Say:

‘Stop now please and close your booklet’

You should intervene if candidates continue after this point.

COLLECT THE ANSWER SHEETS AND THE TEST BOOKLETS,ENSURING THAT ALL MATERIALS ARE RETURNED (COUNTBOOKLETS AND ANSWER SHEETS)

Then say:

‘Thank you for completing the Critical Reasoning TestBattery.’

eqBEFORE STARTING THE QUESTIONNAIRE


The instructions below should be read out verbatim and the same scriptshould be followed each time the GRT1 is administered to one or morecandidates. Instructions for the administrator are printed in ordinary type.Instructions designed to be read aloud to candidates are a larger typefacewith speech marks.

If this is the first or only questionnaire being administered, give an introduc-tion as per or similar to the following example (prepare an amendment if notadministering all tests in this battery):

‘From now on, please do not talk among yourselves, butask me if anything is not clear. Please ensure that anymobile telephones, pagers or other potential distractions areswitched off completely. We shall be doing three tests:verbal, numerical and abstract reasoning. The tests take 8,10 and 10 minutes respectively to complete. During the testI shall be checking to make sure that you are not makingany accidental mistakes when filling in the answer sheet. Iwill not be checking your responses to see if you areanswering correctly or not.’




Then ask:

‘Has everyone got two sharp pencils, an eraser, some roughpaper and an answer sheet.’


‘Print your surname, first name and title clearly on the lineprovided and indicate your title, sex and age by ticking theappropriate boxes. Please insert today’s date which is [ ]’

GRT1 ADMINISTRATIONINSTRUCTIONS

erWalk around the room to check that the instructions are being followed.







‘This test is designed to assess your understanding of wordsand relationships between words. Each question has sixpossible answers. One and only one is correct in each case.Mark your answer by filling in the number box on youranswer sheet that corresponds to your choice. You nowhave a chance to complete the four example questions onPage 3 in order to make sure that you understand the test.

Please attempt the example questions now, marking youranswers in boxes E1 to E4.’

Indicate section

While the candidates are doing the examples, walk around the room to checkthat everyone is clear about how to fill in the answer sheet. Make sure thatnobody is looking at the actual test items during the example session. Whenall have finished (allow a maximum of two minutes) give the answers asfollows:

‘The answer to Example 1 is number 2, sick means thesame as ill.

The answer to Example 2 is number 3, you drive a car andfly an aeroplane.

The answer to Example 3 is number 5, wood is the odd oneout.

The answer to Example 4 is number 4, as both heavy andlight have a relationship to weight.

Is everyone clear about the examples?’

esThen say:

‘REMEMBER:

Time is short, so when you begin the timed test, work asquickly and as accurately as you can.

If you want to change an answer, simply erase your firstchoice and fill in your new answer.

There are a total of 30 questions and you have 8 minutes inwhich to answer them.

If you reach the end before time is called you may reviewyour answers if you wish.








‘Stop now please and turn to Page 12.’

NB: If this is the final test to be used in this battery, instead of the above line,please turn to the instructions in the Ending Test Session box on page 52 ofthis document. If you are skipping a test, please find the appropriate testbookmark for your next test in the margin of the page and replace the aboveline as necessary.


Then say:

‘We are now ready to start the next test. Has everyone stillgot two sharpened pencils, an eraser, some unused roughpaper?’

etIf not, rectify, then say:


Indicate section

Check for understanding, then remembering to read slowly and clearly, go tothe front of the group and say:

‘Please ensure that you are on Page 12 of the booklet andfollow the instructions for this test as I read them aloud.’

Pause to allow page to be found.

‘This test is designed to assess your ability to work withnumbers. Each question has six possible answers. One andonly one is correct in each case. Mark your answer byfilling in the appropriate box that corresponds to yourchoice on the answer sheet.’

You now have a chance to complete the four example questions on Page 13in order to make sure that you understand the test. Please attempt theexample questions now, marking your answers in the example boxes.

Indicate section


‘The answer to Example 1 is number 5, the sequence goesup in twos.

The answer to Example 2 is number 4, as all other frac-tions can be reduced further.

The answer to Example 3 is number 2, 100 is 10 times 10.

The answer to Example 4 is number 5, the journey willtake 1 hour and 30 minutes.


fkThen say:

‘Time is short, so when you begin the timed test work asquickly and as accurately as you can.

If you want to change an answer, fully erase your firstchoice and fill in your new choice of answer.

There are a total of 25 questions and you have 10 minutesin which to attempt them.

If you reach the end before time is called you may reviewyour answers to the numerical test if you wish, but do notgo back to the verbal test.




Deal with any questions, appropriately, then, starting stop-watch or setting acount-down timer on the word BEGIN say:

‘Please turn over the page and begin’



‘Stop now please and turn to Page 20’


You should intervene if candidates continue after this point.Then say:


flIf not, rectify, then say:


Indicate section




‘In this test you will have to work out the relationshipbetween abstract shapes and patterns.

Each question has six possible answers. One and only one iscorrect in each case. Mark your answer by filling in theappropriate box that corresponds to your chosen answer onyour answer sheet. You now have a chance to complete thethree example questions on Page 21 in order to make surethat you understand the test.

Please attempt the example questions now, marking youranswers in the example boxes.’

Indicate section

While the candidates are doing the examples, walk around the room to checkthat everyone is clear about how to fill in the answer sheet. Make sure thatnobody is looking at the actual test items during the example session. Whenall have finished, (allow a maximum of two minutes) give the answers asfollows:

‘The answer to Example 1 is number 5, as the series alter-nates between 2 and 4 squares as does the direction of thetwo squares which return to their original position.

The answer to Example 2 is number 4, as all of the otheroptions have an open side to one of the boxes.

The answer to Example 3 is number 6, as this is a mirrorimage of the pattern.


fmThen say:

‘Time is short, so when you begin the timed test, work asquickly and as accurately as you can.



If you reach the end before time is called, you may reviewyour answers to the abstract test, but do not go back to theprevious tests.









Say:




Then say:

‘Thank you for completing the Graduate Reasoning Test.’

fnBEFORE STARTING THE QUESTIONNAIRE








Then ask:



‘Print your surname, first name and title clearly on the lineprovided and indicate your title, sex and age by ticking theappropriate boxes. Please insert today’s date which is [ ].’


foWalk around the room to check that the instructions are being followed.









Indicate section





The answer to Example 4 is number 5, as dark means theopposite of light.


fpThen say:

‘REMEMBER:















Then say:


fqIf not, rectify, then say:


Indicate section




‘This test is designed to assess your ability to understandnumbers and the relationship between numbers. Eachquestion has six possible answers. One and only one iscorrect in each case. Mark your answer by filling in theappropriate box that corresponds to your chosen answer onthe answer sheet.


Indicate section





The answer to Example 4 is 5, the journey will take 1 hour30 minutes.


frThen say:















Then say:


fsIf not, rectify, then say:


Indicate section




‘In this test you will have to work out the relationshipbetween shapes and figures.



Indicate section






ftThen say:













Say:




Then say:

‘Thank you for completing the General Reasoning Test.’

6REFERENCESBinet. A (1910) Les idées modernessur les enfants Paris: E. Flammarion.

Cronbach L.J. (1960) Essentials ofPsychological Testing (2nd Edition)New York: Harper.

Galton F. (1869) Heriditary GenuisLondon: MacMillan.

Gould, S.J. (1981). The Mismeasureof Man. Harmondsworth, Middlesex:Pelican.

Heim, A.H. (1970). Intelligence andPersonality. Harmondsworth,Middlesex: Penguin.

Heim, A.H., Watt, K.P. andSimmonds, V. (1974). AH2/AH3Group Tests of General Reasoning;Manual. Windsor: NFER Nelson.

Jackson D.N. (1987) User’s Manualfor the Multidimensional AptitudeBattery London, Ontario: ResearchPsychologists Press.

Johnson, C., Blinkhorn, S., Wood, R.and Hall, J. (1989). ModernOccupational Skills Tests: User’sGuide. Windsor: NFER-Nelson.

Budd R.J. (1991) Manual for theClerical Test Battery: Letchworth,Herts UK: Psytech InternationalLimited

Budd R.J. (1993) Manual for theTechnical Test Battery: Letchworth,Herts UK: Psytech InternationalLimited

Stern W (1912) PsychologischeMethoden der Intelligenz-Prüfung.Leipzig, Germany: Barth

Terman, L.M. et. al., (1917). TheStanford Revision of the Binet-Simonscale for measuring intelligence.Baltimore: Warwick and York.

Watson & Glaser (1980) Manual forthe Watson-Glaser Critical ThinkingAppraisal Harcourt BraceJovanovich: New York

Yerkes, R.M. (1921). Psychologicalexamining in the United States army.Memoirs of the National Academy ofSciences, 15.

INTERNATIONAL

general,critical &graduateTEST BATTERYadministration instructions

2instructions

BEFORE STARTING THE QUESTIONNAIRE








Then ask:



‘Print your surname, first name and title clearly on the lineprovided and indicate your title, sex and age by ticking theappropriate boxes. Please insert today’s date which is [ ]’


3instructions










Indicate section





The answer to Example 4 is number 4, as both heavy andlight have a relationship to weight.


4instructions

Then say:

‘REMEMBER:
















5instructions



Indicate section




‘This test is designed to assess your ability to work withnumbers. Each question has six possible answers. One andonly one is correct in each case. Mark your answer byfilling in the appropriate box that corresponds to yourchoice on the answer sheet.’


Indicate section





The answer to Example 4 is number 5, the journey willtake 1 hour and 30 minutes.


6instructions

Then say:













NB: If this is the final test to be used in this battery, instead of the above line,please turn to the instructions in the Ending Test Session box on the finalpage of this document. If you are skipping a test, please find the appropriatetest bookmark for your next test in the margin of the page and replace theabove line as necessary.



7instructions



Indicate section




‘In this test you will have to work out the relationshipbetween abstract shapes and patterns.



Indicate section






8instructions

Then say:













Say:




Then say:

‘Thank you for completing the Graduate Reasoning Test.’

15instructions

BEFORE STARTING THE QUESTIONNAIRE








Then ask:



‘Print your surname, first name and title clearly on the lineprovided and indicate your title, sex and age by ticking theappropriate boxes. Please insert today’s date which is [ ].’


16instructions










Indicate section





The answer to Example 4 is number 5, as dark means theopposite of light.


17instructions

Then say:

‘REMEMBER:















Then say:


18instructions



Indicate section




‘This test is designed to assess your ability to understandnumbers and the relationship between numbers. Eachquestion has six possible answers. One and only one iscorrect in each case. Mark your answer by filling in theappropriate box that corresponds to your chosen answer onthe answer sheet.


Indicate section





The answer to Example 4 is 5, the journey will take 1 hour30 minutes.


Then say:















Then say:


19instructions

20instructions



Indicate section




‘In this test you will have to work out the relationshipbetween shapes and figures.



Indicate section






Then say:













Say:




Then say:

‘Thank you for completing the General Reasoning Test.’

21instructions

reasoning tests technical manual

Documents