raven's apm international technical manual

8/10/2019 Raven's Apm International Technical Manual

1/50


2/50

Copyright 2011 NCS Pearson, Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means,

electronic or mechanical, including photocopy, recording, or any information storage and retrieval system,

without permission in writing from the copyright owner.

Pearson, the TalentLens, Watson-Glaser Critical Thinking Appraisal, and Ravens Progressive Matrices

are trademarks in the U.S. and/or other countries of Pearson Education, Inc., or its affiliate(s).

Portions of this work were previously published.

Produced in the United Kingdom


3/50

Contents

Chapter 1Introduction.......................................................................................................1

Development of the 23-Item Form....................................................................................................2

Internal Consistency Reliability...........................................................................................................3

Content Validity.....................................................................................................................................3

Convergent Validity ..............................................................................................................................3

Criterion-Related Validity...................................................................................................................4

Equivalency Information ......................................................................................................................5Global Applicability...............................................................................................................................7

Development of Ravens APM International Versions ..................................................................7

Chapter 2

Australia/New Zealand (English) ...................................................................9

Translation/Adaptation Process.................................................................................................................9

Sampling Procedure ......................................................................................................................................9

Item/Test Difficulty .......................................................................................................................................11

Distribution of Scores ................................................................................................................................11

Evidence of Reliability .................................................................................................................................12

Chapter 3

France (French)........................................................................................................13

Translation/Adaptation Process................................................................................................................13

Sampling Procedure ....................................................................................................................................13

Item/Test Difficulty .....................................................................................................................................14

Distribution of Scores ...............................................................................................................................15

Evidence of Reliability ................................................................................................................................15


4/50

Chapter 4

India (English) .........................................................................................................17

Translation/Adaptation Process.......................................................................................................17

Sampling Procedure ............................................................................................................................17

Item/Test Difficulty .............................................................................................................................18

Distribution of Scores .......................................................................................................................19

Evidence of Reliability ........................................................................................................................19

Chapter 5

The Netherlands (Dutch) ...............................................................................21

Translation/Adaptation Process...........................................................................................................21

Sampling Procedure ................................................................................................................................21

Item/Test Difficulty .................................................................................................................................23

Distribution of Scores ...........................................................................................................................23

Evidence of Reliability ............................................................................................................................20

Chapter 6

The UK (English) .................................................................................................25

Translation/Adaptation Process...........................................................................................................25

Sampling Procedure ................................................................................................................................25

Item/Test Difficulty .................................................................................................................................27

Distribution of Scores ...........................................................................................................................28

Evidence of Reliability ............................................................................................................................28

Chapter 7

The US (English) ................................................................................................29

Sampling Procedure ..............................................................................................................................29

Item/Test Difficulty ...............................................................................................................................30

Distribution of Scores .........................................................................................................................31

Evidence of Reliability ..........................................................................................................................31


5/50

Appendix ABest Practices in Administering and Interpreting the APM .................................33

Administrators Responsibilities ........................................................................................................33

Assessment Conditions .......................................................................................................................33

Answering Questions ..........................................................................................................................34

Administration.......................................................................................................................................34

Understanding the Scores Reported ...............................................................................................35

Maintaining Security of Results and Materials ................................................................................35

Sources of Additional Best Practice Information...........................................................................35

Instructions for Administering the APM Online............................................................................37

APM Short Test Administration Instructions Paper and Pen..................................................41

References ..........................................................................................................43

Tables1.1 Descriptive Statistics of the APM by Test Version and Administration Order....................6

1.2 Reliability Estimates by APM Test Version and Administration Order .................................6

1.3 Descriptive Statistics and Internal Consistency Reliability Estimates for Ravens

APM Across Countries........................................................................................................................7

2.1 Demographic Information for the Australia/New Zealand Sample .......................................10

2.2 Ravens APM Item Analysis Information for the Australia/New Zealand Sample ................11

2.3 Distribution of APM Scores in the Australia/New Zealand Sample ......................................11

2.4 Internal Consistency Reliability Estimates in the Australia/New Zealand Sample..............12

3.1 Demographic Information for the France Sample .....................................................................13

3.2 Ravens APM Item Analysis Information for the France Sample..............................................14

3.3 Distribution of APM Scores in the France Sample ....................................................................15

3.4 Internal Consistency Reliability Estimates in the France Sample ...........................................15

4.1 Demographic Information for the India Sample.........................................................................17

4.2 Ravens APM Item Analysis Information for the India Sample .................................................18

4.3 Distribution of APM Scores in the India Sample .......................................................................19

4.4 Internal Consistency Reliability Estimates in the India Sample...............................................19

5.1 Demographic Information for the Netherlands Sample ..........................................................22


6/50

5.2 Ravens APM Item Analysis Information for Netherlands Sample .........................................23

5.3 Distribution of APM Scores in the Netherlands Sample .........................................................23

5.4 Internal Consistency Reliability Estimates in the Netherlands Sample ................................24

6.1 Demographic Information for the UK Sample ..........................................................................26

6.2 Ravens APM Item Analysis Information for the UK Sample...................................................27

6.3 Distribution of APM Scores in the UK Sample .........................................................................28

6.4 Internal Consistency Reliability Estimates in the UK Sample.................................................28

7.1 Demographic Information for the US Sample ...........................................................................29

7.2 Ravens APM Item Analysis Information for the US Sample....................................................30

7.3 Distribution of APM Scores in the US Sample ..........................................................................31

7.4 Internal Consistency Reliability Estimates in the US Sample.................................................. 31


7/50

Copyright 2011 NCS Pearson, Inc or its affiliate(s)

1

Chapter 1

Introduction

The Ravens Progressive Matrices have been used in many countries for decades as a measure of

problem-solving and reasoning ability (Raven, Raven, & Court, 1998a). The various versions of the

Ravens Progressive Matrices have been studied in over 45 countries on samples totalling over 240,000

participants (Brouwers, Van de Vigver, & Van Hemert, 2009).

This manual describes the adaptation/translation of the latest 23-item version of the Ravens Advanced

Progressive Matrices (APM) for the US, Australia/New Zealand, France, India, the Netherlands and the

UK. From an international perspective, several enhancements were made to facilitate cross-country

score comparisons, and to standardise the testing experience for administrators and participants. These

enhancements include:

Use of a uniform test format and common test content across countries

Uniform scoring and reporting of scores across countries

Availability of local manager norms for each country, based on a common definition of manager

across countries

Implementation of a common set of items and administration time across countries (i.e., 23

items; 40 minutes).

Description of the Ravens Advanced Progressive Matrices

The Ravens Advanced Progressive Matrices (APM) is a nonverbal assessment tool designed to measure

an individuals ability to perceive and think clearly, make meaning out of confusion and formulate new

concepts when faced with novel information. The APM score indicates a candidates potential for

success in such positions as executive, director, general manager, or equivalent high-level technical or

professional positions in an organisation. These categories of positions typically require high levels of


8/50


2

clear and accurate thinking, problem identification, holistic situation assessment, and evaluation of

tentative solutions for consistency with all available information. Each item in the APM comprises a

pattern of diagrammatic puzzles with one piece missing. The candidates task is to choose the correct

missing piece from a series of possible answers.

Development of the Current 23-Item Form

The current revision of the APM was undertaken to provide customers with a shorter version of the

assessment that maintains the essential nature of the construct being measured and the psychometric

features of the assessment. The APM is a power assessment rather than a speeded assessment, even

though it has a time limit. Speeded assessments are typically composed of relatively easy items and rely

on the number of correct responses within restrictive time limits to differentiate performance among

candidates. In contrast, the APM items have a wide range of difficulty and a relatively generous time

limit, which makes it a power assessment. The 42-minute administration time for the current APM (40

minutes for 23 operational items in Part 1; 2 minutes for 2 experimental items in Part 2) maintains the

APM as an assessment of cognitive reasoning power rather than speed.

N.B. The paper and pencil format does not contain Part 2 experimental items

Classical Test Theory (CTT) and Item Response Theory methodologies were used in the analyses of the

APM data for item selection. Specifically, for each of the 36 items in the previous APM version, the

following indices were examined to select items: item difficulty index (pvalue), corrected item-total

correlation, IRT item discrimination (a) parameter, and IRT item difficulty (b) parameter. Because the

APM was designed to differentiate among individuals with high mental ability, less discriminating items

were dropped from the current version of the APM. For the current APM revision, data were used

from 929 applicants and employees in a number of positions across various occupations. These

individuals took the APM within the period May 2006 through to October 2007. Five hundred and nine

of these individuals provided responses about their current position levels (e.g., Executive, Director,


9/50


3

Manager, and Professional/Individual Contributor). See the Appendix of separate document APM

Development 2007, for more details regarding the composition of the sample.

Internal Consistency Reliability

The internal consistency reliability estimate (split-half) for the APM total raw score was .85 in the U.S

standardisation sample (n=929). This reliability estimate for the 23-item version of the APM indicates

that the total raw score on the APM possesses good internal consistency reliability. Internal consistency

reliability estimates for each country-specific Manager norm group in the global data-collection effort are

summarised in each country-specific chapter within this manual.

Content Validity

In an employment setting, evidence of content validity exists when an assessment includes a

representative sample of tasks, behaviours, knowledge, skills, abilities, or other characteristics necessary

to perform the job. Evidence of the content-related validity of the APM should be established by

demonstrating that the jobs for which the APM is to be used require the problem solving skills

measured by the assessment. Such evidence is typically documented through a thorough job analysis.

Convergent Validity

Evidence of convergent validity is provided when scores on an assessment relate to scores on other

assessments that claim to measure similar traits or constructs. Years of previous studies on the APM

support its convergent validity (Raven, Raven, & Court, 1998b). In a sample of 149 college applicants,

APM scores correlated .56 with math scores on the American College Test (Koenig, Frey, & Detterman,

2007). Furthermore, in a study using 104 university students, Frey and Detterman (2004) reported that

scores from the APM correlated .48 with scores on the Scholastic Assessment Test (SAT).

Evidence of convergent validity for the current version of the APM is supported by two findings. First, in

the standardisation sample of 929 individuals, scores on the current APM correlated .98 with scores on


10/50


4

the previous APM. Second, in a subset of 41 individuals from the standardisation sample, the revised

APM scores correlated .54 with scores on the Watson-Glaser Critical Thinking AppraisalShort

Form (Watson & Glaser, 2006).

Criterion-Related Validity

Criterion-related validity addresses the inference that individuals who score better on an assessment

will be more successful on some criterion of interest (e.g., job performance). Criterion-related validity

for general mental ability tests like the APM is supported by validity generalisation. The principle of

validity generalisation refers to the extent that inferences from accumulated evidence of criterion-

related validity from previous research can be generalized to a new situation. There is abundant

evidence that measures of general mental ability, such as the APM, are significant predictors of overall

performance across jobs. For example, in its publication on the Principles for the Validation and Use of

Personnel Selection Procedures, SIOP (2003) notes that validity generalisation is well-established for

cognitive ability tests. Schmidt & Hunter (2004) provide evidence that general mental ability predicts

both occupational level attained and performance within one's chosen occupation and does so better

than any other ability, trait, or disposition and better than job experience (p. 162). Prien, Schippmann,

and Prien (2003) observe that decades of research present incontrovertible evidence supporting the

use of cognitive ability across situations and occupations with varying job requirements (p. 55). Many

other studies provide evidence of the relationship between general mental ability and job performance

(e.g., Kolz, McFarland, & Silverman, 1998; Kuncel, Hezlett, & Ones, 2004; Ree & Carretta, 1998; Salgado,

et al., 2003; Schmidt & Hunter, 1998; Schmidt & Hunter, 2004).

In addition to inferences based on validity generalisation, studies using the APM over the past 70 years

provide evidence of its criterion-related validity. For example, in a validation study of assessment

centres, Chan (1996) reported that scores on the Ravens Progressive Matrices correlated with ratings

of participants on initiative/creativity (r=.28, p< .05). Another group of researchers (Gonzalez,


11/50


5

Thomas, and Vanyukov, 2005) reported a positive relationship between scores on the Ravens APM and

performance in decision-making tasks. Fay and Frese (2001) found that APM scores were consistently

and positively associated with an increase of personal initiative over time (p. 120). Recently, Pearson

(2010) conducted a study of 106 internal applicants for management positions in which APM scores

were positively correlated with trained assessor ratings of thinking, influencing, and achieving. In

addition, manager applicants scoring in the top 30% of APM scores were 2-3 times more likely to

receive above average ratings for the Case Study/Presentation Exercise, Thinking Ability, and

Influencing Ability than applicants in the bottom 30% of APM scores.

The APM Manual and Occupational Users Guide (Raven, 1994; Raven, Raven, & Court, 1998b) provide

additional information indicating that the APM validly predicts the ability to attain and retain jobs that

require high levels of general mental ability. The validity information presented in this manual is not

intended to serve as a substitute for locally obtained validity data. Local validity studies, together with

locally derived norms, provide a sound basis for determining the most appropriate use of the APM.

Therefore, users of the APM should study the validity of the assessment at their own location or

organisation.

Equivalency Information

Occasionally, customers inquire about the equivalence of online versus paper administration of the APM.

Studies of the effect of the medium of test administration have generally supported the equivalence of

paper and computerised versions of non-speeded cognitive ability tests (Mead & Drasgow, 1993). To

ensure that these findings held true for the Ravens Advanced Progressive Matrices, Pearson TalentLens

conducted an equivalency study using paper-and-pencil and computer-administered versions of the test.

The psychometric properties of the two forms were compared to determine whether the mode of

administration impacted scores and whether decision consistency could be assured across modes of

administration.


12/50


6

In this study, a counter-balanced design was employed using a sample of 133 adult participants from a

variety of occupations. Approximately half of the group (n=68) completed the paper form followed by

the online version, while the other participants (n=65) completed the tests in the reverse order. The

interval between test sessions ranged from 13 to 91 days (M=22.9, SD=12.1). Table 1.1 presents means,

standard deviations, and correlations obtained from an analysis of the resulting data. Analyses of the test

modes revealed that there was no significant difference in the examinees raw scores on the APM

between paper (M= 11.9, SD= 4.9) and online versions (M= 11.7,SD= 4.8), t(132) = -0.95, p= .34). The

total scores from the different versions were highly correlated (r= .78).

Table 1.1

Descriptive Statistics of the APM by Test Version and Administration OrderAPM Test Version

Paper OnlineAdministrationOrder N M SD M SD rPaper First 68 12.4 4.8 13.0 5.5 .85Online First 65 11.5 4.8 10.3 3.7 .73Total 133 11.9 4.8 11.7 4.9 .78

Table 1.2 displays the reliability estimates of the paper and online versions from the different test

administration groups. These estimates demonstrate that reliability estimates of APM scores fall within

the acceptable range regardless of test modes or administration order, providing additional support for

equivalence.

Table 1.2 Reliability Estimates by APM Test Version and Administration OrderAPM Test Version

Paper OnlineAdministration Order rsplit ralpha rsplit ralphaPaper First .88 .83 .88 .87Online First .86 .82 .75 .70Total .86 .83 .85 .83


13/50


7

Global Applicability

The nonverbal aspect of the APM minimizes the impact of language skills on performance on the

assessment. The fact that the Ravens shows less influence of cultural factors than more verbally-laden

assessments has made it very appealing as global measure of cognitive ability. The global exposure of the

Ravens abstract reasoning format has several important advantages for inclusion in a global selection

strategy. Specifically, its familiarity increases the likelihood of local management support, it promotes

positive participant reactions and helps ensure that scores arent unduly influenced by language or

culture (see Ryan & Tippins, 2009 for more information on implementing a consistent and effective

global selection system). The following chapter provides important information in helping multinational

companies incorporate the APM into a global selection system, and to make informed comparisons of

applicants performance on the APM across countries and cultures.

Development of Ravens Advanced Progressive Matrices (APM) International Versions

Development for each country-specific version of the APM followed a uniform process focused on

adapting and translating the test instructions in a way that ensured consistent measurement of the

construct across countries. The international versions are based on the same 23-item set, including the

practice items, matrix stimuli, and response options. Table 1.3 presents a summary of results by country,

including the number of managers tested, characteristics of the score distribution, and total score

reliability.

Table 1.3 Descriptive Statistics and Internal Consistency Reliability Estimates forRavens APM Across Countries

Country N Mean SD Skewness Kurtosis ralpha rsplitAustralia/NewZealand 128 11.95 4.17 0.02 -0.39 .77 .78France 106 14.33 4.09 -0.34 -0.09 .74 .79India 100 9.51 4.22 0.19 -0.80 .79 .82Netherlands 103 13.01 4.53 -0.10 -0.64 .81 .83UK 101 12.38 4.72 0.04 -0.71 .83 .85US 175 12.23 4.14 -0.03 -0.13 .77 .81


14/50


8

Results showed that internal consistency reliability estimates across countries were adequate (e.g., rsplit=

.79-.85), and that sample homogeneity and differences in prior exposure to cognitive ability testing may

account for observed differences in raw score means across countries. Detailed information on the

collection and analyses of country-specific norms is provided throughout the remainder of this Manual.


15/50


16/50


10

Table 2.1

Demographic Information for the Australia/New Zealand SampleN Percent

128 100.0Education Level

Year 11 or equivalent 2 1.6

Year 12 or equivalent 4 3.1Certificate III / IV 2 1.6Diploma 3 2.3Advanced Diploma 2 1.6Bachelor 28 21.9Graduate Certificate 1 0.8Graduate/PostgraduateDiploma 20 15.6Master 47 36.7Doctorate 3 2.3Other 4 3.1Not Reported 12 9.4

SexFemale 61 47.7Male 55 43.0Not Reported 12 9.4Age

21-24 1 0.825-29 9 7.030-34 19 14.835-39 25 19.540-49 28 21.950-59 29 22.7

60-69 5 3.9Not Reported 12 9.4Years in Occupation

15 18 14.1Not Reported 14 10.9


17/50


18/50


12

Evidence of Reliability

Split-half (rsplit), Cronbachs alpha (ralpha), and standard error of measurement (SEM) were calculated using

the Australia/New Zealand Sample. Results are presented in Table 2.4. Internal consistency reliability

estimates were consistent with the values found in the US standardisation sample and confirm that the

APM demonstrates adequate reliability in the Australia/New Zealand sample.

Table 2.4

Internal Consistency Reliability Estimates in the Australia/New Zealand SampleRavens APM N rsplit ralpha SEMTotal Score 128 .78 .77 1.96Note. SEMwas calculated based on split-half reliability.


19/50


20/50


14

Item/Test Difficulty

Classical Test Theory (CTT) and Item Response Theory (IRT) methodologies were used in the analysis

of the APM data collected in France. Specifically, for each of the 23 items in the APM, the following

indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item

discrimination (a) parameter, item difficulty index (pvalue), and item-total correlation. Results are

presented in Table 3.2.

Table 3.2 Ravens APM Item Analysis Information for the France Sample

APM ItemNumber

ItemDifficulty (b)Parameter

(IRT)

Item-AbilityCorrelation

(IRT)Discrimination (a)Parameter (IRT)

Item DifficultyIndex (pvalue;

CTT)Item-Total

Correlation (CTT)

1 -3.38 0.05 0.90 0.97 0.002* - - - - -3 -1.04 0.34 0.97 0.80 0.254 -1.97 0.32 1.03 0.90 0.265 -0.76 0.39 1.03 0.76 0.326 -0.45 0.15 0.45 0.71 0.027 -0.70 0.32 0.90 0.75 0.238 -0.70 0.45 1.13 0.75 0.379 -0.07 0.44 1.10 0.64 0.3410 -0.63 0.49 1.21 0.74 0.4311 0.34 0.38 0.88 0.56 0.2612

-0.17 0.48 1.21 0.66 0.4013 -0.07 0.25 0.55 0.64 0.1314 0.49 0.50 1.32 0.53 0.4115 0.98 0.40 0.92 0.43 0.2716 1.18 0.42 0.99 0.39 0.3017 0.04 0.52 1.35 0.62 0.4418 0.34 0.57 1.56 0.56 0.5019 0.78 0.24 0.31 0.47 0.1320 0.93 0.45 1.08 0.44 0.3221 0.88 0.43 1.03 0.45 0.3222 0.93 0.48 1.20 0.44 0.3823 3.06 0.32 0.98 0.12 0.24

*100 percent of the sample obtained a perfect score on Item 2, making the IRT and CTT values inestimable.


21/50


15

Distribution of Scores

Characteristics of the distribution of APM raw scores for the France sample are provided in Table 3.3.

Table 3.3 Distribution of APM Scores in the France SampleN Minimum Maximum M SD Skewness Kurtosis

100 3 22 14.3 3.9 -0.31 -0.16



the France Sample. Results are presented in Table 3.4. Internal consistency reliability estimates were

consistent with the values found in the US standardisation sample and confirm that the APM

demonstrates adequate reliability in the France sample.

Table 3.4 Internal Consistency Reliability Estimates in the France SampleRavens APM N rsplit ralpha SEMTotal Score 100 .79 .74 1.80Note. SEMwas calculated based on split-half reliability.


22/50


23/50


17

Chapter 4

India (English)

Translation/Adaptation Process

Instructions for the APM were reviewed and adapted by a team of test-development experts from the

Pearson TalentLens Bangalore, India office.

Sampling Procedure

The Bangalore, India office of Pearson TalentLens recruited and administered the online version of the

APM to 100 manager-level examinees across various industries. These individuals took the APM under

timed (40-minutes) and proctored (i.e. supervised) conditions within the period February, 2010 through

to April, 2010. Table 4.1 provides the demographic data for the final sample of N=100.

Table 4.1

Demographic Information for the India SampleN Percent


10th 1 1.012th 3 3.0Bachelors 49 49.0Masters 39 39.0Doctoral 2 2.0Other 6 6.0Sex

Female 22 22.0Male 78 78.0Age

20-24 2 2.025-29 27 27.030-34 33 33.035-39 17 17.040-44 10 10.045-49 4 4.0Not Reported 7 7.0


24/50


18



of the APM data collected in India. Specifically, for each of the 23 items in the APM, the following indices

were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item discrimination (a)

parameter, item difficulty index (pvalue), and item-total correlation. Results are presented in Table 4.2.

Table 4.2 Ravens APM Item Analysis Information for the India Sample

APM ItemNumber


(IRT)



Item Difficulty Index (pvalue; CTT)

Item-TotalCorrelation

(CTT)

1 -2.85 0.39 1.05 0.86 0.302 -2.27 0.53 1.21 0.79 0.43

3 -1.85 0.54 1.21 0.73 0.464 -2.05 0.43 1.01 0.76 0.315 -1.49 0.53 1.17 0.67 0.436 -1.43 0.52 1.13 0.66 0.427 -1.21 0.46 0.96 0.62 0.348 0.12 0.40 0.83 0.37 0.319 -0.57 0.57 1.29 0.50 0.4810 -0.57 0.52 1.14 0.50 0.4311 0.64 0.55 1.26 0.28 0.4812 0.01 0.38 0.73 0.39 0.2713 0.52 0.43 0.98 0.30 0.3514 1.42 0.29 0.88 0.17 0.2215 1.34 0.33 0.93 0.18 0.2516 1.59 0.23 0.83 0.15 0.1517 0.17 0.40 0.82 0.36 0.3018 0.76 0.30 0.74 0.26 0.2019 0.52 0.51 1.16 0.30 0.4320 0.83 0.45 1.07 0.25 0.3721 1.42 0.32 0.94 0.17 0.2622 1.03 0.35 0.91 0.22 0.2623 3.94 0.01 0.90 0.02 -0.02


25/50


26/50


20

Ravens Advanced Progressive Matrices (APM)


27/50


21

Chapter 5

Netherlands (Dutch)

Translation/Adaptation Process

Instructions for the APM were translated into Dutch by an independent translator, contracted by the

Amsterdam, Netherlands office of Pearson TalentLens. Two Dutch-speaking test-development experts

from the Pearson TalentLens Amsterdam office reviewed the translation and refined the final translation.

Sampling Procedure

The Amsterdam, Netherlands office of Pearson TalentLens recruited and administered the online

version of the Dutch APM to 138 manager-level examinees across various industries. These individuals

took the APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the period

September through to October 2009.Thirty-five participants were eliminated from the sample after a

review of self-reported job titles revealed that they did not qualify as a Manager. Table 5.1 provides the

demographic data for the final sample of N=103.


28/50


22

Table 5.1 Demographic Information for the Netherlands SampleN Percent



29/50


23



of the APM data collected in the Netherlands. Specifically, for each of the 23 items in the APM, the

following indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item



Table 5.2 Ravens APM Item Analysis Information for the Netherlands Sample

APM ItemNumber


(IRT)





(CTT)

1 -2.95 0.29 1.02 0.94 0.242 -2.76 0.38 1.09 0.93 0.313 -1.77 0.50 1.19 0.85 0.444 -1.59 0.40 1.03 0.83 0.345 -1.13 0.24 0.68 0.78 0.166 -0.46 0.47 1.06 0.67 0.397 -1.00 0.33 0.82 0.76 0.238 0.32 0.43 0.87 0.52 0.349 0.17 0.50 1.12 0.55 0.4210 -0.58 0.51 1.19 0.69 0.4411 0.17 0.53 1.23 0.55 0.4612

0.67 0.44 0.89 0.46 0.3513 0.17 0.48 1.03 0.55 0.3914 0.37 0.48 1.04 0.51 0.3915 1.42 0.47 1.04 0.32 0.3716 0.88 0.36 0.66 0.42 0.2617 0.98 0.50 1.11 0.40 0.4118 -0.14 0.51 1.15 0.61 0.4319 0.77 0.59 1.38 0.44 0.5220 1.30 0.40 0.84 0.34 0.3021 0.98 0.45 0.94 0.40 0.3822 1.19 0.40 0.84 0.36 0.3023 2.99 0.28 0.92 0.12 0.20


Characteristics of the distribution of APM raw scores for the Netherlands sample are provided in Table

5.3.

Table 5.3 Distribution of APM Scores in the Netherlands SampleN Minimum Maximum M SD Skewness Kurtosis103 2 22 13.0 4.5 -0.10 -0.64


30/50


24



the Netherlands sample. Results are presented in Table 5.4. Internal consistency reliability estimates

were consistent with the values found in the US standardisation sample and confirm that the APM

demonstrates adequate reliability in the Netherlands sample.

Table 5.4 Internal Consistency Reliability Estimates in the Netherlands SampleRavens APM N rsplit ralpha SEMTotal Score 103 .83 .81 1.87Note. SEMwas calculated based on split-half reliability.


31/50


32/50


26

Table 6.1 Demographic Information for the UK SampleN Percent


GCSE equivalent 3 2.97A-level, Scottish Highers or

equivalent

1 0.99

Higher Education Certificateor Diploma

12 11.88

BA 29 28.71BSc 16 15.84BEd 1 0.99LLB 3 2.97Master Degree 26 25.74Doctoral Degree 2 1.98Other 7 6.93Not Reported 1 0.99Sex

Female 46 45.54Male 46 45.54Not Reported 9 8.91Age

16-19 4 3.9620-24 1 0.9925-29 10 9.930-34 28 27.7235-39 31 30.6940-44 12 11.8845-49 5 4.95

50-54 2 1.9855-59 2 1.98Not Reported 6 5.94Years in Occupation


33/50


27



of the APM data collected in the UK. Specifically, for each of the 23 items in the APM, the following

indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item



Table 6.2 Ravens APM Item Analysis Information for the UK Sample

APM ItemNumber


(IRT)





(CTT)

1 -3.12 0.26 1.03 0.94 0.222 -2.94 0.33 1.09 0.93 0.313 -2.27 0.36 1.06 0.88 0.324 -1.06 0.40 0.93 0.73 0.345 -1.47 0.32 0.86 0.79 0.256 -1.06 0.36 0.83 0.73 0.297 -0.93 0.38 0.85 0.71 0.318 0.03 0.55 1.25 0.54 0.509 -0.40 0.58 1.37 0.62 0.5410 -0.08 0.42 0.78 0.56 0.3311 0.45 0.53 1.17 0.46 0.4712

0.51 0.39 0.61 0.45 0.3113 0.51 0.54 1.18 0.45 0.4614 0.56 0.58 1.34 0.44 0.5215 1.00 0.54 1.16 0.36 0.4616 1.42 0.48 0.99 0.29 0.3917 0.45 0.52 1.14 0.46 0.4518 0.24 0.52 1.13 0.50 0.4619 0.67 0.45 0.86 0.42 0.3820 1.35 0.37 0.68 0.30 0.2621 1.35 0.42 0.82 0.30 0.3022 1.06 0.52 1.11 0.35 0.4423 3.72 0.39 0.99 0.06 0.29


34/50


35/50


29

Chapter 7

US (English)

Sampling Procedure

The US sample consisted of 175 manager-level examinees across various industries. These individuals

took the online APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the

period April through to August 2008. Table 7.1 provides the demographic data for the final sample of

N=175.

Table 7.1 Demographic Information for the US Sample

N Percent175 100.0

Education Level

1-2 Years of College 1 0.6Associates 1 0.63-4 Years of College 4 2.3Bachelors 65 37.1Masters 68 38.9Doctorate 6 3.4Sex

Female 39 22.3

Male 107 61.1Not Reported 29 16.6Age

21-24 2 1.125-29 17 9.730-34 23 13.135-39 28 16.040-49 53 30.350-59 20 11.460-69 2 1.1Not Reported 30 17.1Years in Occupation

1-2 years 7 4.02-4 years 19 10.94-7 years 32 18.37-10 years 19 10.910-15 years 16 9.115+ years 19 10.9Not Reported 33 18.9


36/50


37/50


31


Characteristics of the distribution of APM raw scores for the US manager sample are provided in Table

7.3.

Table 7.3 Distribution of APM Scores in the US SampleN Minimum Maximum M SD Skewness Kurtosis175 2 23 12.2 4.1 -0.03 -0.13



the US sample. Results are presented in Table 7.4. Internal consistency reliability estimates confirm that

the APM demonstrates adequate reliability in the US sample.

Table 7.4 Internal Consistency Reliability Estimates in the US SampleRavens APM N rsplit ralpha SEMTotal Score 175 .81 .77 1.80Note. SEMwas calculated based on split-half reliability.


38/50


32



39/50


33

Appendix A

Best Practices in Administering and Interpreting the APM

Administrators Responsibilities

The best way for administrators to prepare for the assessment is to take it themselves, complying with

all directions. The administrator should ensure that the organisations assessment process complies with

professional standards and practices, including Human Resources policies. Before candidates take the

assessment, the administrator should explain the nature of the assessment, why it is being used, the

conditions under which the candidates will be assessed, and the nature of any feedback they will receive,

as determined by organisational policy.

Though not required for job applicants in all countries, we recommend obtaining informed consent from

the candidate before the assessment is taken. An informed consent form is a written statement

explaining the type of assessment instrument to be administered, the purpose of the evaluation, and

who will have access to the data. The candidates signature validates that he or she has been informed of

these specifics. Administering the APM takes about one hour in total, including giving directions to

candidates, answering questions about the assessment procedures, and actual assessment time .

Assessment Conditions

The following conditions are suggested for improving score accuracy and maintaining the cooperation of

the candidates: good lighting; comfortable seating; adequate desk or table space; comfortable positioning

of the computer screen, keyboard and mouse, when administering online; a pleasant and professional

attitude on the part of the administrator; and freedom from noise and other distractions.


40/50


34

Answering Questions

Though the instructions for completing the assessment online are presented on-screen, it is important

to develop and maintain rapport with candidates. The administrator is responsible for ensuring that

candidates understand all requirements and interact with the assessment interface appropriately.

Candidates may ask questions about the assessment before they begin. Clarification of what is required

of candidates and confirmation that they understand these requirements is appropriate. See the section

in Appendix A Instructions for Administering the APM Online for an appropriate script when starting

the assessment. Paper and pencil format APM Short Test Administration Instructions are also

provided in Appendix A.

If any candidates have routine questions after the assessment has started, try to answer the questions

without disturbing the other candidates. However, if candidates have questions about the interpretation

of an item, they should be encouraged to respond to the item as they best understand it.

Administration

Both online and paper & pencil formats require supervised administration and begin with a set of four

practice items with an answer and explanation. Although un-timed, allow up to three minutes for

completion of the practice set. Online test takers have 40 minutes to complete all 23 items in Part 1.

Part 1 automatically goes into time out at the end of 40 minutes. Test takers have 2 minutes to

complete the 2 items in Part 2. Part 2 automatically goes into time out at the end of 2 minutes. During

each part of the assessment, test takers have the option of skipping items and returning to them later if

time remains. If test takers finish Part 1 of the assessment before the 40-minute time limit has expired,

they may review their answers, or move on to Part 2. Please note that the Part 2 experimental items

are not included in the paper and pencil format.

If a test takers computer develops technical problems during the assessment, the administrator should

move the candidate to another suitable computer location if possible and log back into the system as


41/50


35

before. If the technical problems cannot be solved by moving to another computer location, the

administrator should contact Pearsons TalentLens Technical Support for assistance.

At the end of the assessment session, thank each candidate for his or her participation and check the

computer station to ensure that the assessment is closed. Note that scoring will not occur and the

assessment will stay in In Progress status until the candidate has completed the assessment.

Understanding the Scores Reported

The online interpretive report includes a total raw score as well as a percentile score corresponding to

the total raw score. The percentile score is a standardised score that indicates the standing of the

participant relative to individuals in the norm group. The percentile score indicates the proportion of

the norm group who possess less of the ability than the participant. For example, if a participants APM

score is at the 75thpercentile of a given norm group, it means that the participant scored higher than or

equal to 75% of the people in the norm group.A score above the 90thpercentile is considered well

above average in comparison to the norm group, above the 70thabove average, above the 30thaverage

and above the 10thbelow average. Scores at the 10thpercentile or lower are considered well below

average.

Maintaining Security of Results and Materials

APM scores are confidential and should be stored in a secure location accessible to authorised

individuals only. It is unethical as well as poor assessment practice to allow assessment score access to

individuals who do not have a legitimate need for the information. The security of assessment materials

(e.g. access to online assessments) and protection of copyright must also be maintained by authorised

individuals.


42/50


36

Sources of Additional Best Practice Information

Governmental and professional regulations cover the use of all personnel selection procedures in most

countries. Relevant source documents that the user may wish to consult are the International

Guidelines For Test Use (International Test Commission, 2000), the Code of Good Practice for

Psychological Testing (British Psychological Society, 2012),

Psychological Testing: A User's Guide (British Psychological Society, 2012) and, Data Protection and

Privacy Issues Relating to Psychological Testing in Employment-Related Settings (British Psychological

Society, 2012).

For local statutes and legal proceedings that influence an organisations equal employment opportunity

obligations, the user is referred to their local governing authority that monitors employment selection

practices.


43/50


37

Instructions for Administering the APM Online

Once your PPU survey/assessment is created, copy the PPU URL to the browser of the computer(s) tobe used for supervised testing.

If testing is to be anonymous, test takers will be taken to the beginning of the survey/assessment. If test

takers are required to self-register for the survey/assessment, they will see the following page.

Supervised Test Administrator Introduction:

After a short welcome introduction say:

To sign on please enter you name and your email address in the boxes provided, and then

click submit.

When all candidates have signed on say:

The onscreen directions will take you through the entire process, which begins with a

welcome page and some general questions about you. At the end of the test there are a

few more general questions.

While all candidates are completing the general questions ask:

Do you have any questions before you click next to start the assessment? (Ensure all

candidates have completed the onscreen general questions and are ready to begin the

test)

(c) Pearson Education Ltd


44/50


38

All surveys/assessments begin with a generic welcome instruction

Pre assessment general questions all fields are required.


45/50


39

Back end general questions are all optional. All data is anonymous and strictly confidential.


46/50


40



47/50


41

APM Short Test Administration Instructions Paper and Pencil

If more than 10 people are to be tested in a group, it is necessary to have one assistant for every 10-15 people.

The test administrator needs: these instructions, a stopwatch, and one copy of the Test Booklet and Record Form.

Each person taking the test needs: two pencils, eraser and one copy of the Test Booklet and Record Form.

After your informal introduction to the test session:

DO Hold up the Test Booklet and a Record Form as you;

SAY For this session there is one Test Booklet and a Record Form on which to record your answers. No marks shouldbe made on the Test Booklets.

Please look at the Advanced Progressive Matrices Record Form. Please complete your name, todays date, your jobtitle, the industry you are currently employed in, years in position and highest level of education you havecompleted.

To monitor fairness in testing, equal opportunities information is also requested. Completion of the equal

opportunities section at the bottom of this page is optional. All of the groups presented here are protected by UKequality and discrimination laws. A prefer not to say option is available for each question if you do not wish todisclose this information. Choosing this option will have no impact on your score or your standing in the selectionprocess.

DO Allow time for candidate/s to complete the details on the cover of the record form.

SAY This is a test of observation and clear thinking. The first part of the test contains 4 practice items followed byexplanations of the answers to them. This is intended to show you how the test works, or, if you have seen tests ofthis sort before, to remind you how they work.

DO Hold up a Test Booklet

SAY Please open your Test Booklet to the first page and read the instructions but do not turn the page yet.

DO Allow time for the candidates to read the instructions, then:

SAY Now turn the page. You will see that this is Practice Item 1.

Now turn over your Record Form.

You will see that under the heading Practice Items there is a column of numbers 1, 2, 3, and 4. This is where theanswers go.

DO Hold up a Record Form and point to Column 1.

SAY Now look back at your Test Booklet. The top part of Practice Item 1 is a pattern with a piece missing. Look at thepattern, think what the piece needed to complete the pattern correctly both along and down must be like. Then findthe right piece out of the eight options shown below. Please look at this practice item and try to solve the problem.

Record your answer by placing a single line across the number of the answer you think is correct in the appropriatebox on the Record Form.

DO Pause for approximately 1 minute to allow candidates to look at the first item.

SAY Now turn the page and we will read through the answer together.

Number 8 is the correct answer because it is the only piece that correctly completes the pattern going across the

row and down the column.

Numbers 1, 2, and 6 complete the pattern of one solid line going down the column, but do not complete thepattern of three dotted lines going across the row.


48/50


42

Numbers 4 and 5 correctly complete the pattern of three dotted lines going across the row, but do not completethe pattern of one solid line going down the column.

Numbers 3 and 7 do not complete the pattern of three dotted lines going across the row, and do not complete thepattern of one solid line going down the column.

Number 8 is the only answer that works going across the row and down the column.

DO Tester and assistants check that everyone has correctly marked 8 for Practice Item 1 on their Record Form.

SAY This is a Practice Set. It is not important to complete all the items. The important thing is to notice how theproblems develop and to learn how to solve them. Continue with the rest of the Practice Items on your own.Record your answers by making a single horizontal mark through the number of the answer you think is correct onyour Record Form. Then review the explanations to the answers.

If you make a mistake or want to change your answer, put a cross, or X, through your incorrect answer, and thenput a single line through the correct answer. Do not try to erase the incorrect answer.

Now complete Practice Items 2-4 by yourselves. You will have up to 3 minutes for this. When you have finished donot turn the page until you are instructed to do so.

DO Allow 3 minutes unless it is clear that test takers have all finished before this.

Tester and assistants check that everyone is putting their answers in the correct column.

SAY Please stop now. Shortly we will start the real test. The items in it are similar to those you have just completedexcept that there are more of them, and they get progressively more difficult.

As a general guideline, the correct response will always fit across each row and down each column. You can lookalong each row and down each column to help you determine the missing piece. The correct option will match thepattern going across the row and down the column.

As you did when completing the Practice Items, record your answers to the test by making a single horizontal markthrough the number of the answer you think is correct on your Record Form under Test Items.

DO Hold up a Record Form and show where the Test Items begin.

SAY Please do not mark the Test Booklet. You will be allowed 40 minutes to complete the test. Remember it isaccurate work that counts. Attempt each item in turn. Do your best to find the correct option before you go on tothe next item. If you get stuck, move on and come back to the item later. Remember, however, that you may findthe next item harder and it may take you longer to check your answers carefully. Are there any questions?

DO Pause briefly.

SAY Now turn the page in your Test Booklet to the first Test item.

DO Pause briefly. Check that everyone is ready to start.

SAY Begin now.

DO Start timing 40 minutes.

At the end of 40 minutes:

SAY Everyone stop working, please. Close your Test Booklet. Please check that you have put your name, as well astodays date, on the Record Form.

DO Check that they have put their names on their Record Forms, and filled in the additional details.At the end of the session thank everyone for their time and tell them what use will be made of their data. If appropriate,explain what arrangements will be made for them to find out their results. It would also be constructive to reassure testtakers regarding the confidentiality of test scores.


49/50


43

References

American Educational Research Association, American Psychological Association, & National Council ofMeasurement in Education. (1999). Standards for educational andpsychological testing. Washington, DC:Author.

Brouwers, S. A., Van de Vijver, F. J. R., & Van Hemert, D. A. (2009). Variation in Ravens ProgressiveMatrices scores across time and place. Learning and Individual Differences,19, 330338.

Chan, D. (1996). Criterion and construct validation of an assessment centre.Journal of Occupational and Organizational Psychology, 69, 167181.

Fay, D., & Frese, M. (2001). The concept of Personal Initiative: An overview of validity studies.Human Performance, 14(1), 97124.

Frey, M. C., & Detterman, D. K. (2004). Scholastic assessment or g? The relationship between theScholastic Assessment Test and general cognitive ability. Psychological Science,15(6), 373378.

Gonzalez, C., Thomas, R. P., & Vanyukov, P. (2005). The relationship between cognitive ability anddynamic decision making. Intelligence, 33, 169186.

International Test Commission (2000). International guidelines for test use. Downloaded electronicallyfrom www.intestcom.org/itc_projects.htm

Koenig, K. A., Frey, M. C., & Detterman, D. K. (2008). ACT and general cognitive ability.Intelligence, 36, 153160. doi:10.1016/j.intell.2007.03.005

Kolz, A. R., McFarland, L. A., & Silverman, S. B. (1998). Cognitive ability and job experience as predictorsof work performance. The Journal of Psychology, 132(5), 539548.

Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity,and job performance: Can one construct predict them all?Journal ofPersonality and Social Psychology, 86(1), 148161.

Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive abilitytests: A meta-analysis. Psychological Bulletin, 114(3), 449458.

Pearson (2010).Management assessment process: Preliminary report.Unpublished manuscript 34 Copyright 2011 NCS Pearson, Inc. All rights reserved.

Prien, E. P., Schippmann, J. S., & Prien, K. O. (2003).Individual assessment as practiced in industry and consulting. Mahwah, NJ: Erlbaum.

Raven, J. (1994). Occupational users guide: Ravens Advanced Progressive Matrices & Mill Hill Vocabulary Scale.Oxford, UK: Oxford Psychologists Press Ltd.

Raven, J., Raven, J. C., & Court, J. H. (1998). Raven manual: Section 4, Advanced Progressive Matrices,1998edition. Oxford, UK: Oxford Psychologists Press Ltd.


50/50

Ree, M. J., & Carretta, T. R. (1998). General cognitive ability and occupational performance. In C. L.Cooper & I. T. Robertson (Eds.), International review of industrial andorganizational psychology(Vol. 13, pp. 159184) Chichester, UK: Wiley.

Ryan, A.M., & Tippins, N. (2009). Designing and implementing global selection systems.Chichester, UK: Wiley-Blackwell.

Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & De Fruyt, F. (2003). International validitygeneralisation of GMA and cognitive abilities: A European community meta-analysis.Personnel Psychology, 56, 573605.

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnelpsychology: Practical and theoretical implications of 85 years of research findings.Psychological Bulletin, 124, 262274.

Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainmentand job performance.Journal of Personality and Social Psychology, 86(1), 162173.

Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use ofpersonnel selection procedures (4th ed.). Bowling Green, OH: Author.

U.S. Department of Labor. (1999). Testing and assessment: An employers guide to good practices.Washington, DC: Author.

Watson, G., & Glaser, E. M. (2006). Watson-Glaser Critical Thinking AppraisalShort Form manual. SanAntonio, TX: Pearson.

raven's apm international technical manual

Documents