why use psychometrics for employee selectionjp

Compiled by Joseph Paulson [email protected] 09448270474

1/18

Why use Psychometrics to Support Employee Selection and Development?

Introduction

Ever since I have completed my Level A&B Certification from BPS, people have asked me no end of questions about what psychometrics is, what it does, whether it works, and is it ethical. I would like to use this opportunity to share my learned perspectives on these issues for the benefit of those who seek more information.

Psychometrics has been a controversial subject for many recent years, yet it is becoming more and more popular with more users and more and more publishers entering the fray. My experience with psychometrics starts in 1979 when I went for my selection in AFSB Mysore to be selected as pilot in IAF. I joined the same Board as an assessor administering PABT in 1995 and was associated with the PABT research by DIPR till 2003. My next step was to do Level A& B Certification from BPS in Dec 2005 and there after MBTI certification in Dec 2006.

As a commoner, all of you will have read articles in the press and will probably have completed questionnaires in magazines/on the internet. Some of you may have seen quiz shows on TV or attempted Psychometrics as part of career Counselling or some selection process or the other. Your impressions will be based on your own experiences of being assessed. All of these experiences will almost certainly have affected your lives in important ways. Hence, any person in the street is unlikely to be neutral about the use of psychometric tests.

On the basis of all this, how is one likely to react when invited to take a psychometric test as part of a recruitment programme? Well, almost certainly be apprehensive. If it is an ability test that one is about to take, one will worry about not doing well. First one will think back to previous tests that one has done in the past. One will begin to consider how one should prepare for the test. Many sources are often available for such preparation although not all are helpful. I once read a book, intended to provide useful advice to job applicants. To the question "What do I need to do to do well on an ability test?" the reply was "Ability tests are not difficult, all you need is a quick mind and a good intelligence". I think you would agree that this is not particularly helpful.

For the candidate asked to take a personality test, the concerns will be different. One may think "Does this mean that they will be able to read my mind, to know what I'm thinking?" One will think about aspects of own character that might make oneself unsuitable for the job. And one will wonder about how much they could actually find out. And, of course, one’s attention will turn to ways in which one might be able to cover some things up. One will ask oneself what he should do to present oneself in the best light, and what one should say or not say.

Once the whole assessment process has been completed, whether one succeeds or fails in one’s job application, one will be left with many questions unanswered. Did one fail because one was too honest? Can psychometric testing truly allow for one’s individuality? Did the assessment fail to take account of the fact that English was the second language? Did one succeed just through luck or because one cheated? And how did one’s score compare with that of other candidates? And these concerns are not just the candidate’s concerns, but also those of the selection panel and, ultimately, of the psychometricians who design and maintain the tests and questionnaires themselves.

I shall Endeavour to address some of these questions by first addressing the question "What is psychometrics?" I shall then go on to consider common misconceptions about psychometrics, followed by an examination of current debates that involve psychometric issues. I shall conclude by addressing the key question of this lecture "Does psychometrics have a part to play in employee selection?"


2/18

What is psychometrics?

So what exactly is psychometrics? Psychometrics has been defined as "The science of psychological assessment". The four major principles of psychometrics are reliability, validity, freedom from bias, and standardization.

Reliability

Reliability is the extent to which an assessment is free from error. There are several different types of reliability. Inter-rater reliability examines the level of agreement between two raters on the same assessment. This might be an observation of behaviour in assessment centre/interview, or marks given for an in-tray exercise where the candidate was asked to draft a letter to a client marks/interpretation in any projective tests. It would be foolish to make use of an assessment if there was evidence that raters were unable to agree on the quality of the candidates' performance.

Another type of reliability is test-retest reliability. Here the same test is given to the same group of candidates on more than one occasion, usually within 6 months and the consistency of scores across these occasions is examined. If there is little consistency then this tells us that we can have little confidence in the results. There is simply no point in using a test that gives a completely different score every time we use it - no different really from weighing yourself every morning and getting a different reading on the scales when you know that it is extremely unlikely that you have lost or gained weight. As a psychometric test undergoes development its reliability is monitored closely and the test adjusted to keep this value to a maximum.

Validity

Validity is the extent to which a test or questionnaire assesses what it purports to assess. There are four main types of validity. These are face validity, content validity, predictive validity and construct validity. Face validity exists if the candidate and all involved see the assessment as appropriate. For example, a personality test, designed for clinical use that contained the question "Do you hear voices" would lack face validity in most recruitment situations. The candidate may feel that that this is an inappropriate question to ask of someone who is applying for a job. A student once described face validity as "giving a test to a group of people and looking at the expressions on their faces". Not a bad definition really.

With content validity we match the test specification, the blueprint that underlies the development of the test, to the job description for the post for which the person is applying. If the two do not match then the use of the test would in be inappropriate for the job in question. Predictive validity concerns the availability of evidence that the test will be effective in predicting how well a person will perform if appointed. This is the prime objective for most psychometric tests used in staff selection and promotion.

Finally, construct validity assesses the extent to which the assessment relates to its construct. Constructs are ideas like 'emotional intelligence' or 'IQ'. Simply describing a test as a test of emotional intelligence is not enough. We need to be able to present arguments and convince others that 'emotional intelligence' is a real entity and say exactly what we mean by it. We also need to show how the items we have chosen to test emotional intelligence are actually measuring this construct and not something else.

An assessment can be reliable but not valid

Handwriting analysis can be quite reliable in that two handwriting experts trained in the same system may have a high level of agreement about what is meant by a particular aspect of handwriting. That is, they may both believe that if handwriting leans backwards this means that the writer has a yearning to return to things as they were in the past. But this interpretation may be wrong. It could well be that there are other


3/18

reasons why peoples' writing leans backwards. Maybe it has something to do with whether a person is right or left handed? In this case we would say that while handwriting analysis can be reliable in that it produces consistent reports from people trained in its methods, this does not necessarily mean that it has any validity whatsoever for the purpose of assessing personality.

IQ testing I would now like to take a moment to address the controversial issue of the construct validity of the intelligence test, more commonly known as the IQ test. The first laboratory for psychometrics was set up by Sir Francis Galton at the Grand Exhibition of 1883 held in South Kensington. Sir Francis was the first person to attempt to measure intelligence psychometrically. He offered to test the faculties of people who were visiting the exhibition, and made a charge of three pence per person for this service. But what is intelligence? Edwin Boring in 1923 gave a particularly unconvincing argument for the construct validity of intelligence when he defined it as "That which is measured by intelligence tests".

In fact, psychologists use the term 'intelligence' very sparingly, if at all. It is a very old word carrying with it rather too much conceptual baggage. Intelligent or 'gifted' individuals were seen by the Ancient Greeks as having received 'a gift from the gods". Since the time of Darwin it is more likely to be seen as a gift from nature, but a gift nonetheless. And if you have not received this gift, the worse for you - you can't fight god and you shouldn't fight Mother Nature. Such a belief undermines the evidence that people can change, can learn, and can increase their ability to do any number of things.

Another reason that psychologists and psychometricians are so cautious about the use of intelligence testing is its murky role in the eugenics movement of the last century. Before 1940, IQ tests were routinely used not just to assess special educational needs but also to provide evidence for sterilization programme for those who were believed to be 'subnormal'. This abuse of the science is something that most of us hoped had been buried by World War II.

Freedom from bias

I would like to turn now to the third psychometric principle, freedom from bias. Test bias arises if the test is unfair to certain groups, that is, if one consequence of using the test is a disproportionate representation of people of a particular background, or of men or women, in the workforce. When this happens the test is described as having 'adverse impact'. There are several ways in which a psychometric test might be biased. It may be that some of the items contain bias. An item may contain very colloquial words or phases the meaning of which may elude speakers of English as a second language. Irony and sentences such as "He goes round with a chip on his shoulder' should be avoided as they can cause confusion.

Standardization Finally, standardization refers to the availability of appropriate comparison groups, so that the score of an individual can be compared with the average for the population - this is called 'norm referencing' - or interpreted alongside those of other people in a similar job; called criterion referencing. Before we can make sensible use of any test we need to know what a particular score means. If someone is told that they have achieved a score of 75 on a test they might initially be quite pleased - until they are told that everyone else who applied for the job scored over four hundred and fifty. Also, whatever anyone else may have scored on a test, we as practitioners need to know the sort of performance we can reasonably expect from a person with this particular score.


4/18

Questions for recruiters

Anyone involved in staff selection, whether using psychometric tests or not, should be able to satisfy themselves that their assessment procedures have complied with the four psychometric principles. Also, they should be in a position to provide a justification of their choice of assessment procedures to others if called upon to do so. They need to know that their assessment system is reliable in that it can be depended upon to produce a consistent result. They also need to know that it is valid in that the person selected is likely to be the best person for the job in comparison to all the other candidates. They will also need to be sure that their selection procedure is free from bias. Finally, they need to be sure that the standards they have set are the correct ones. Common misconceptions about psychometrics

I am now going to move on to consider three common misconceptions about psychometrics that underlie many of the questions that I and other psychometricians are asked by clients, students and the interested public. These are the belief that the four psychometric principles only apply to psychometric testing, the belief that it is easy to cheat in a personality test, and the belief that psychometric tests fail to take account of a person's true individuality.

The misconception that psychometric principles only apply to psychometric tests

The first common misconception is that the four psychometric principles of reliability, validity, standardization and freedom from bias may be right for psychometric tests and questionnaires, but that they do not apply to other forms of assessment such as school examinations or the more open-ended type of interview. This is not true. They apply to any assessment. In terms of reliability it would be counter-productive to reject a psychometric test on the grounds that it was not reliable, if instead selection was by an interview that proved to be even less reliable. Or, in relation to the psychometric principle of freedom from bias, it would be inappropriate to reject a psychometric test on the grounds that it discriminated against a particular ethnic group, if selection were to be based instead on a procedure that was even more biased.

An obvious alternative to a psychometric test of ability might be the use of high school/college academic grades. But this will also have its problems. In the sense in which the term 'bias' is used in psychometrics, school/college results are very biased indeed!

With respect to the psychometric principle of standardization, it has recently been suggested that psychometric tests might be a more effective technique for selecting students for higher institutes of learning as the selection system itself has not paid sufficient attention to the need for standardization. Universities have always had pre-existing ideas of how students with particular grades might perform as learners. Employers may begin to suspect that today's graduates are not as effective as were previous graduate recruits. Finally, when it comes to the psychometric principle of validity, there are many cases in employee selection where psychometric tests have been rejected on the grounds of suspected poor validity, only to be replaced by alternatives such as a particular type of interview, without any objective consideration of whether these alternatives are themselves valid. The need to demonstrate validity should be just as important for an interview as it is for a psychometric test.

I have talked so far only about the application of psychometric principles to recruitment situations. However, there are many other forms of assessment, such as the driving test, licensing qualifications for doctors, for accountants and for other professionals, psychiatric diagnoses, and the assessment of special educational needs in children, to which they equally apply. What I would like to emphasize here is that, whatever you


5/18

may think of psychometric tests themselves, the psychometric principles are inescapable in any assessment process. Far more training is needed in the application of these principles in all areas of assessment. And applicants with psychometric experience are increasingly being sought by recruitment consultancies and executive search organizations, as well as by human resources departments. The misconception that it is easy to cheat on personality tests

The second misconception I would like to address is the widespread belief that it is easy to get a 'good' result on a personality test by faking answers to questions. Now, of course, there can be no doubt that it is possible to lie when answering a question in a questionnaire. But this applies when answering a question in an application form, or in an interview. In all these cases, the recruiter needs to be aware of this possibility and to have well developed strategies for dealing with it. Hence, when a test is constructed, precautions will normally be taken and safeguards built into the test itself.

Many questionnaires contain a lie detector. There will be, scattered among all the other items, a series of items such that people will only tend to answer them in a particular way when they are lying. An example of such an item might be "It always pays to tell the truth (Agree or Disagree)". Clearly the statement cannot be true, certainly not 'always' as the question specifically states. Most people will reluctantly disagree with the statement. Some may agree because at first glance it seems to them to be true, perhaps because this is what they were taught at Primary School. But the majority who agree will do so because they think that 'Agree' is the desired answer, even though they do not seriously believe it to be true. Of course, any one item like this on its own could not predict lying. But if the same pattern was observed for a whole series of lie-detector items then we would probably feel rather hesitant about accepting the candidate's answers to the real questions. This is only one of a number of strategies to detect lying that are used by psychometricians when constructing a test. Hence, as with an interview, people may lie, but by carefully choosing which questions to ask, by ordering the questions in a particular way, and by setting internal alarms matched to certain patterns of answers, liars should know that their subterfuge may well be detected.

So failure to detect dishonest responding is not the most important issue when considering the relative merits of personality questionnaires and interviews as assessment techniques. There are more important ones. An advantage of the personality questionnaire is that it puts exactly the same questions to large numbers of people in different situations. In this way we build up a detailed knowledge of how people generally respond to particular questionnaire items and combinations of items. This provides a background against which an individual person's responses can be interpreted. But the questionnaire does not have the ability of the interviewer to react to the applicant, to tailor the questions to the facts as they emerge. In the meantime the personality questionnaire can be a valuable addition to the selection process. Anything we can do to supplement the interview is to be welcomed for, as most of you will know, in many situations the interview is not particularly reliable, or valid, or standardized or free from bias.

The misconception that psychometric tests fail to take account of a person's individuality

A third common misconception is the widespread belief that the people who design and work with psychometric tests are only interested in numbers and fail to understand the human side of assessment and evaluation. This belief is also shared by some social scientists, who see psychometrics as purely quantitative, while interviews and other 'soft' techniques of assessment are believed to be more qualitative. I believe that rather too much is made of the distinction between the qualitative and the quantitative approaches. However unstructured an interview, it must always result in a quantitative ranking of candidates in terms of perceived suitability for the post, otherwise no appointment could be made. Similarly, psychometric tests, whatever mathematical techniques are used in


6/18

their construction, must have a qualitative aspect. They must take account of the real-world situation in which people live.

For academic reasons some people may want to draw a sharp line between qualitative and quantitative methods, but in practice this is usually unhelpful. A prudent human resource professional will ignore neither aspect of the assessment process. Of course, psychometrics as a discipline has always been back-footed in the quantitative / qualitative debate by its very name 'psycho-metrics'. The 'metric' part refers to measurement, surely the epitome of quantitative analysis. And many psychometricians have taken pleasure in quoting a phrase from William Thompson, also known as Lord Kelvin, who was the arch-measurer of the 19 th Century and was responsible for the Kelvin scale of temperature. He said "Everything that exists must exist in some quantity and can therefore be measured".

Today, however, psychometrics is no longer 'the science of psychological measurement'. It is now 'the science of psychological assessment' and there is a subtle but important difference between the two. Often, but not always, we can obtain someone's measure by summing up their positive attributes, subtracting their negative attributes and examining this total, rather like the scales of justice. But the whole can be more than the sum of the parts, as perhaps can best be demonstrated by what in psychometrics is referred to as the cloning problem. Suppose we are able to identify the perfect personality profile for 'a good salesman'. When we use this profile in staff selection we will be able to 'clone' this ideal profile and select a group of good salesman. But will this be a good team? Probably not, because teams need diversity. The simple addition of attributes fails to recognize that there are many different types of good salesman. There are many different profiles that fit the bill. In this way the 'quantitative' psychometric approach appears to fall down when compared with the more 'qualitative' interview. But does psychometrics have to be about simply totaling up scores? Not any more. In the past, tests and questionnaires were necessarily limited to paper and pencil, but today questions can be presented on computer screen. The technology for doing this is complex and state-or-the-art, but a great deal of progress has already been made. Computer-assisted testing programs are in widespread.

A questionnaire can be treated like an expert system. This is a term used in artificial intelligence research to describe the type of computer program that has largely replaced the role of the traditional bank manager in making decisions on who should or should not be allowed a loan. Such programs can also make quite a good job of medical diagnosis on health websites. Most of you probably know of them in the form of the 'wizards' (for example that annoying paper-clip) that give advice when you use a computer.

More advanced artificial intelligence systems are likely to find increased utility in biometrics and psychometrics alike. Neural networks can learn to improve their performance on the basis of their successes and failures at hitting targets Once a target had been set, such as only selecting staff who would stay in a job for at least 6 months, a neural network could learn to predict this outcome from the patterning of answers given in personality tests at selection. With the advent of the internet this presents both problems and promise. The problems mostly involve human rights issues, the right to privacy in particular, but after some initial stumbling with the implementation of the provisions of the Data Protection Act/RTI, most organizations are taking seriously and coming to grips with the issues involved. The huge growth in spam, not just emails but faxes and now telephone calls as well, does somehow focus the mind!

The speed at which the information revolution is moving is now beginning to break down barriers between adjacent disciplines, both in academia and in practice. Is the 'lie detector' a biometric or a psychometric device? Is a psychometric test of achievement a psychometric or an educational test? Is the use of keyboard behaviour to detect fraud psychometrics or is it actuarial science? And, if we link assessments of psychological constructs such as fear and greed to the behaviour of the stock market, is this psychometrics or econometrics? Maybe in all these cases, it is both. It's not just politics


7/18

and the media that have been thrown into turmoil by the communications revolution. We are moving into a new era in which psychometrics and its fellow sciences will inevitably play an increasing role.

The promises are many and varied, so I will focus on just one, the potential for improved prediction. Applied psychometrics is, in essence, a predictive technology, an actuarial science. We use selection tests to predict how candidates would perform if given the job, and we select those who best fit this prediction. All actuarial sciences are currently undergoing revolutionary changes, whether it is the prediction of investor behaviour, life expectancy, insurance or credit risk. Actuarial prediction is a necessary part of all these functions, just as it is for human resources. But the enormous advances made have not been without their problems. New Artificial Intelligence systems probably had a role in several of the stock market crashes of the late 1990s; the use of postcodes is raising a number of new issues; as is genetic testing for insurance purposes. Hence we cannot expect the use of predictive technology in human resources to escape controversy. But neither can we bury our heads in the sand. The future is after all inescapable. Many of the mistakes occurred not because people took unnecessary risks, but because they failed to foresee the consequences of the new technology. Inaction also holds risks. Most personnel directors can think of occasions where a failure to predict the outcome of a particular appointment has proved disastrous and anything we can do to improve our predictions must be a major priority.

Application of psychometrics to On-line testing The next issue I want to talk about demonstrates the need to be prepared for the unexpected. The internet, in particular, continually throws up surprises. It is probably already obvious to all here that on-line internet assessment is here to stay and will continue to evolve. A recent survey has shown that 70% of corporations use tests to measure human performance and nine out of ten are going to be using more tests in 2004. Already, many of these tests are delivered over the web. In the meantime on-line sales, on-line banking, and on-line booking are now starting to rise exponentially.

While more research is needed to demonstrate that on-line testing is as good as or better than paper and pencil testing, for many practical applications this need has simply being overtaken by events. When an organization is recruiting for posts that demand computer proficiency, and if the necessary tests are available on computer, it often makes no sense to use paper. Indeed as fewer and fewer people are proficient at handwriting or managing without a spelling checker and a wizard, a new concern is emerging. Do paper and pencil test adversely impact computer users?

Does psychometrics have a place in employee selection?

I would now like to come back to the main question of this lecture "Does psychometrics have a place in employee selection?" I am sure you will not be surprised to hear that my answer is "yes" .. But you might perhaps like some more details.

Let me give you a practical example. Psychometric tests can be classified into two broad categories: those that assess ability and those that assess personality and these can be used separately or together depending on the particular position to be filled. Let's say that you want to recruit a manager in a company. Any of the following three tests would give you a standardized assessment to produce an overall profile: 15FQ+, OPP and SPI along with GRT 2 which is an ability test.

Human Resource Professionals need to be qualified in order to administer psychometric tests. Both psychologists and non-psychologists can obtain these qualifications. A Level A certificate is required for ability testing, and a Level B certificate for personality testing. There are various agencies offering training courses leading to both of these qualifications. The British Psychological Society is the issuing body for these certificates.


8/18

Ten ways that psychometrics can help you:

First, it can provide an efficient, standardized assessment of an individual's strengths and capabilities. Moreover, if psychometric tests or questionnaires have been administered prior to an interview, they can provide information concerning each candidate's likely strengths and possible weaknesses, and these can be pursued as part of the interview process.

Second, many psychometric tests come accompanied by an extensive research based literature that can provide justification for their use in a wide variety of situations and settings.

Third, psychometrics can be applied to a recruitment activity of any scale, ranging from a single post to a multi-centre multi-national recruitment campaign carried out world-wide.

Fourth, numbers are no object. We can test a single individual, or we can test millions, as with the minimum competency tests for children used in the US.

Fifth, psychometrics provides a set of principles - the psychometric principles of reliability, validity, standardization and freedom from bias - that form the basis for the objective examination of the merits and demerits of a wide range of different assessment systems.

Sixth, the psychometric principles provide a framework within which the effectiveness of policies such as an organization’s equal opportunities policy can be monitored.

Seventh, psychometrics can be a specialist career path within human resources that is exciting, rewarding and much in demand.

Eighth, knowledge of psychometrics enables human resource executives to increase their influence in the boardroom by being at the forefront of new developments in IT and the use of the web.

Ninth, the possession of psychometric expertise provides an opportunity for human resources to influence management decisions through the presentation of scientific evidence that can demonstrate links between recruitment policy and company success.

And tenth, because the area is so fast moving, psychometrics offers those organizations involved a competitive advantage over their less reactive counterparts.


9/18

More on psychometric testing

The term 'psychometric' comes from two Greek words and, literally translated, means 'measurement of the mind'. The term 'measurement', as used here, is not meant to imply that mental abilities or qualities can be measured in the same way as physical quantities. Rather, it is meant to convey the fact that psychometrics involves a 'scientific' approach to the assessment of these abilities and qualities. The best way to illustrate this is by an example from the assessment of personality.

The most commonly used method of assessing personality in employment situations is subjective judgement. Thus, when a job candidate is being interviewed, the interviewer tries to make a judgement of what the person is like as a person and how he or she might behave in a variety of different situations. This judgement will of course be influenced by many factors - for example:

• How experienced the interviewer is - not just in interviewing applicants but in interacting with people and making predictions about how they are likely to behave

• The extent to which the candidate is projecting his/her true self or projecting an image that he/she feels will go down well with the interviewer

• The range of situations covered by the interview and the extent to which these are relevant to behaviour in a work situation

• Who precisely the interviewer is comparing the candidate with - with the interviewer him/herself, with the interviewer's friends and associates, with people who work in the same company, with people who have attended for similar interviews in the past?

• What particular prejudices the interviewer may have - for example, does he/she believe that women are not well suited to jobs involving authority over others or that men are incapable of working in a caring role?

• What particular internal 'theories of personality' does the interviewer hold dear - for example, does he/she believe that if someone gives a firm handshake, they must be a strong character?

• What sort of mood and general state of health the interviewer happens to be in that day

• What sort of mood and general state of health the candidate happens to be in that day

This is not to say that some interviewers may not be able to make very accurate judgments of personality. Probably some are. The problem is that we don't really know who these people are and, even worse, they probably don't even know themselves.

In contrast, the psychometric approach to the assessment of personality tries as far as possible to overcome the sources of error and likely misjudgment mentioned above. There are several different psychometric approaches to the measurement of personality, but they all have the following elements in common.

• They are based on scientific research (at least in so far as it is possible to be scientific in relation to human behaviour)

• They used a standardized approach to obtaining the relevant information - i.e. every person who takes a particular personality questionnaire will be asked precisely the same set of questions. This differs markedly from the interview situation where each candidate may well be asked very different questions (even when the interview is a 'structured'


10/18

interview). Effectively this means that the psychometric approach gives each person equal opportunity to show what they are like across the same range of behaviour, attitudes and interests as everyone else taking the questionnaire.

• They use a standardized approach to judging the information provided by the candidate. Unlike in an interview where a given statement by the candidate might be judged differently by different interviewers (and even judged differently by the same interviewer on different occasions), the psychometric approach evaluates the responses of all candidates according to a fixed set of rules. For each personality questionnaire, there is a scoring key - and the same scoring key is used for each person taking the questionnaire. There can therefore be no argument, at least at this stage in the process, in interpreting particular responses the candidate may have given.

• They use a standardized and quantified approach to expressing the conclusions reached by the assessment. In the case of most psychometric tests, this is done by comparing the scores obtained by a candidate with the scores obtained by a large sample of people who have taken the test on a previous occasion. Thus, in contrast to the interviewer who might at best be able to say "I think this chap is pretty sociable and should be able to get on well with people", the psychometric questionnaire would conclude that, for example, the candidate has scored "at the 75th percentile on the scale of Sociability". What this means is that 75 percent of people in the comparison sample scored at a lower level than this (were less sociable) and only 25 percent of people scored higher (were more sociable). In this way, the psychometric approach is able to make a quantitative assessment of just how sociable the person is, when compared to other people who have taken the test in the past.

• The effectiveness of psychometric tests can be evaluated objectively. For example, studies can be conducted to see whether people vary in their responses to a questionnaire on different occasions. If it is found that their responses vary greatly from occasion to occasion, this means that the test may be unreliable and may need to be improved. Studies can also be conducted to see whether the predictions made from the test or questionnaire is actually born out in practice. For example, the scores on a test of Sales Ability can be correlated mathematically with actual sales figures. If the people who score highly on the test do not turn out to have higher sales figures than people who score low on the test, this suggests that the test is not assessing what it is supposed to assess and should probably be abandoned.

Although most of the above example has been taken from the assessment of personality, these general principles are applicable to all other areas of psychometric testing - testing of ability, of aptitude, of interests, of motivation - and also to other forms of assessment which, although not making use of questionnaires or tests, nevertheless are based on objective and standardized methods of assessing people.

Different types of psychometric tests

The following are the principle types of psychometric tests used in employment settings:

1. Tests of ability

This category covers what psychologists sometimes refer to as 'tests of maximum performance'. Such tests ask a person to do his or her best on a task and their performance is judged in terms of how well they have done - i.e. normally in terms of how many correct answers they gave. Ability tests are sometimes subdivided in to tests of achievement and tests of aptitude. Tests of achievement are intended to assess a person's present capability and tests of aptitude are intended to assess what a person is likely to be good at some time in the future.

The types of ability assessed in employment settings include:

• general mental abilities such as verbal reasoning, numerical reasoning and abstract reasoning


11/18

• more specific job-related abilities such as clerical aptitude, mechanical aptitude and spatial reasoning

Most ability tests provide just a single score, indicating how the person has performed on the ability in question in comparison with people who have taken the test previously.

2. Personality questionnaires

Personality questionnaires differ from ability tests in that there are usually no right or wrong answers to the questions. Rather, the purpose of the questionnaire is to assess the person's typical way of behaving, thinking, feeling or perceiving in particular situations.

Usually, personality questionnaires provide not just one but several scores for the person: one score for each 'scale' or 'dimension' of personality being assessed. The overall pattern of scores provides what is called the 'personality profile and it is this which is used as the basis for making interpretations about how the person is likely to behave in different circumstances.

An important difference between ability tests and personality tests is that considerably more skill and judgement is necessary to interpret the results of the latter. This is the point at which the 'science' ends and the judgement begins.

3. Tests of interests, motivation and attitudes

Although not clearly distinguishable from personality tests, there is another group of tests which attempt to assess other more specific aspects of a person's style and approach to life.

Career interest questionnaires for example assess the extent to which a person shows interests in particular work 'themes' or 'areas' (for example, working outdoors, creative work, working with people) and these, in conjunction with ability tests can be used by employment counselors in offering guidance to individuals on career choice.

Motivation questionnaires attempt to assess what types of motivation are most significant to an individual (e.g. achievement, security, affiliation with others and so on) and these questionnaires are also useful in career planning and employment counseling.

Advantages of psychometric testing

Properly developed psychometric tests and questionnaires, when used by competent and appropriately qualified individuals, have the following advantages:

• They lead to judgments that are likely to be more valid than judgments made by other means. This is the most important advantage of psychometric assessment.

• They are relatively cheap and easy to administer when compared to other approaches. For example, although it may seem relatively expensive for a company to pay for its staff to become qualified in psychometric assessment and then on top of this to pay for the cost of the testing itself, these costs pale into insignificance when one considers just how long it would take to obtain the same information about a person. At the risk of putting it simplistically, it could be argued that the information obtained from a good personality questionnaire might take several months of knowing and working with a person to obtain by other means.

• They are likely to lead to considerable cost-benefits in the long term. Whether it be for selection of new staff or development of existing staff, the expenses involved in psychometric assessment are minimal when compared with the costs of high-turn over, under-performance or mis deployment of staff.


12/18

Disadvantages of psychometric testing

The following are significant dangers associated with psychometric testing

• Despite what has been said in previous sections, there are numerous tests and questionnaires on the market which purport to be 'psychometric instruments' but which are not. Unfortunately, it is very difficult for untrained people to distinguish these from good psychometric instruments. In many cases, these tests and questionnaires have been put together by people with no background in psychometrics and they have very little actual utility and value for the purposes for which they are marketed.

• Lack of correct training is also a significant danger in the use of psychometric testing. Although there exists in the UK a training qualification system developed by the British Psychological Society, it is not uncommon for tests to be used by people who are not adequately trained to use them. Indeed, even attendance at a recognized training course is no guarantee that a person will at all times use tests and questionnaires correctly since some instruments, particularly personality questionnaires, require considerable experience and the possibility of misinterpretation or inappropriate interpretation of results is ever-present.

• Another important danger with psychometric testing is the use of personality questionnaires to try to assess a person's ability or skill in a particular area. For example, if a person scores highly on a personality dimension called 'Leadership', this does not mean that he or she will actually possess a high level of leadership skill. Rather it means that the person has the basic personality characteristics that are commonly found amongst effective leaders and, with sufficient experience and given the development of certain necessary skills, has the potential to become an effective leader. Unfortunately however, scores on scales such as this are often taken to imply that the person already has all the necessary skills and is already capable of performing at a high level in the area in question.

Qualifications needed to use psychometric tests

There are instruments on the market for which you need no specific qualification in order to purchase and use. In general, you should avoid such instruments as they are unlikely to have been developed with the rigor normally required of psychometric tests. In contrast, for most properly developed psychometric tests available in the UK, you will need to be qualified at either 'Level A' or 'Level B'.

Level A and Level B are the two principal stages of training in the certification system for psychometric test training set up by the British Psychological Society. These qualifications are available following attendance at courses offered by the major UK test publishers and by other training providers.

Still more on what is a psychometric test?

A psychometric test is a standardized method for assessing a mental or psychological attribute such as a skill, an ability or a dimension of personality. To say it is 'standardized' means that the assessment is made in a standardized manner (i.e. in the same way for all people and from occasion to occasion), that the person's responses to the test are recorded and scored in a standardized manner (e.g. by the use of a fixed scoring key) and that the results are interpreted in a standardized manner (for example, by comparing the scores obtained by an individual to those obtained by a large sample of people who have taken the test previously).

How can I know if a particular test or questionnaire is worth using?

Firstly you need to know that the test has been developed according to sound psychometric principles and whether there is evidence of its reliability (its ability to give consistent results) and its validity (its ability to assess what it is supposed to assess). This is a complex area and you will probably not be able to determine the answer to


13/18

these questions unless you are a psychologist or have undergone special training in psychometric assessment. However, you should be able to seek advice on these issues from a consultancy firm specializing in psychometric assessment, particularly if the firm employs qualified industrial / occupational psychologists.

Secondly, you need to know whether the test or questionnaire will answer the particular questions you are asking. For example, if you are selecting for a particular position, one of the principal questions you are likely to be asking is "how well would this candidate perform in the job a year from now?” In this case, you would need to know that the test is able to make valid predictions about future performance in work of the type in question. You might also need to know whether the person will work well in a team environment. For this purpose, you would need to know whether the test is able to provide scores on personality dimensions relevant to team work. More than likely, no single test or questionnaire will be able to answer all the questions you need answers to and you will probably need to use a combination of tests and questionnaires, in addition to other selection methods. Your consultancy firm should be able to help you select a battery of techniques suitable for the purpose in question.

What does reliability mean?

Reliability is essentially a measure of a test's ability to produce consistent results. What this means is that if the person were to take the test on several different occasions then, practice effects aside, they would obtain the same result. To say 'practice effects aside' means that one has to take into consideration the obvious point that if a person takes an ability test several times, they are likely to improve their score each time simply due to the effect of practice. When assessing the reliability of a test, psychologists are able to use statistical techniques to control for this effect.

There are different types of reliability and different ways of assessing it. Though each method normally is expressed in terms of an index called the reliability coefficient. This index ranges from 0 to 1 with 0 meaning zero reliability and 1 meaning 100% reliability. The reliability of good ability and reasoning tests is normally somewhere between 0.7 and 0.9. The reliability of personality measures tends to be lower for a variety of reasons, but should desirably be around 0.7 or above. For reputable tests from reputable publishers, you should be able to check the reliability of the test by consulting the test manual or enquiring directly of the test publishers.

Reliability is considered to be an essential characteristic of any test. If a test is unable to measure something consistently, then it is unlikely to lead to valid inferences being drawn about the person being assessed. Nevertheless, reliability by itself is not sufficient. A test must also be valid for the purpose for which it is going to be used and it is validity which is in fact the most important consideration when evaluating a test.

What does validity mean?

The term validity can be defined in two ways: (a) the ability of the test to assess what it is supposed to assess, and (b) the ability of the test to allow valid inferences to be drawn from its results. These are in fact two ways of saying the same thing. For example, suppose we are using a personality questionnaire which contains a dimension called "Social Confidence". We firstly wish to know that the score on this dimension really does reflect the amount of social confidence a person has - or putting it differently, that people with high scores on this dimension really will show more confidence in social settings than people who score low on the dimension. Alternatively, we might say that we want to be confident in any inferences we make about the person on the basis of their score on this dimension. For example, if we wish to make the inference that people who score high on this dimension will perform well in customer service jobs, then we would want to know that this would be a correct inference which would be born out in practice.


14/18

Psychologists measure validity in different ways and refer to different types of validity - though it is not always easy to understand the difference between the different types of validity. We say a test or questionnaire has 'construct validity' if indeed the test measures the psychological attribute (or construct) that it purports to measure. Thus, an intelligence test is said to be valid if it does indeed measure how much intelligence a person has. Unfortunately however, there is a chicken-and-egg problem here in that in order to know if this is the case, we need to have some independent measure of a person's intelligence to check the results of the test against. And then we need to have some further means of assessing the construct validity of that measure. It is a never ending problem.

For this reason, in employment situations (particularly in selection situations), a second form of validity, 'predictive validity' is thought to be more relevant. Predictive validity is the ability of a test or questionnaire to make specific predictions about something which can be objectively assessed. For example, we could assess the predictive validity of a dimension of 'Sociability' by seeing if people who score highly on that scale tend to spend more time in the company of friends and associates than those who score low on the scale. Or for a dimension of 'Sales Potential', we could follow up a group of sales personnel who had been given the questionnaire at selection and see if those who got high scores on the Sales Potential dimension really did have higher sales figures a year later than those how got low scores.

Just as with reliability, validity is an essential characteristic of a test. Before choosing a test to use for a given purpose, you should always check for evidence of its validity. Such evidence can be found either in the test manual or by enquiring directly to the test publisher.

Can psychometric tests replace conventional methods of assessing people?

The answer to this is partly yes and partly no. Properly developed and valid psychometric tests can certainly replace less reliable and valid techniques. For example, unstructured interviews when used to assess both ability and personality are certainly far less effective than psychometric methods. Judgments about a person's ability solely on the basis of their educational qualifications also leave much to be desired and are far better done by means of tests. However, tests and questionnaires can only provide part of the picture. Their principal function is as a supplement to both traditional methods and other objective methods of assessment. A balanced assessment program, depending on the purposes to be achieved, should include structured interviews, tests and questionnaires and objective assessments of behaviour (for example structured exercises, role-plays and other observation methods).

Why use ability tests when a personality test can tell you all you need to know

If a personality test can assess such dimensions as social confidence, leadership, and rationality and so on, why should you need to use ability tests or other assessment methods to measure these characteristics? The problem is that what personality tests assess is not ability but inclination and disposition. If a person scores highly on a personality dimension of rationality, this means not that they are capable of rational thinking but that they have particular personality characteristics which dispose them to a rational rather than an irrational style of thinking. A particular individual might be very highly disposed to such a style of thinking but, because he/she lacks certain intellectual and cognitive skills, actually be extremely poor when it comes to rational thinking. In general therefore, if you need to know how good someone is at doing something, the best approach is to watch them doing it (by using a standardized objective exercise). If such a standardized exercise is not possible, the next best thing is to use a psychometric test designed to assess the ability in question. Personality measures of ability-related dispositions come a very poor third after these.

Is a personality test always better than 'human judgement'?


15/18

Let us imagine that there exist certain individuals who, without having had training in psychology or a related discipline, just happen to have a great deal of insight into people. How would one expect such people to fare in competition with personality questionnaires? Well, they would probably do extremely well. Some people really do have great insight and really could tell you more about a person than a personality questionnaire could. And if your company's chief interviewer happens to be such a person, then you are in luck and you can save a great deal of money on personality testing.

The problem is - how do you know whether your chief interviewer is such a person? Because they themselves say so? A lot of people are extremely confident in their ability as judges of other people, but in most cases their confidence has little to support it. Could you yourself check out their ability as judges of people? Well here you have a problem as the only way you could check out their ability would be by being an expert judge of people yourself. And then someone else is going to have to confirm that you are indeed such an expert - and so the problem goes on.

To cut a long story short, there no doubt are a few people around who can make better judgments of personality than can be made by a personality questionnaire. The problem is that they are probably very few and far between and it is extremely difficult to find out who they are. Much better to give up the hope of finding the Holy Grail and invest some money in getting your people trained in personality assessment.

Having said this however, it nevertheless remains the case that a personality questionnaire by itself can only provide you with a set of scores, one for each dimension assessed by the questionnaire. After that point you really do need to use expert judgement. Some people will naturally be better at this than others, but all will need to be carefully trained in the sorts of inferences one can make from personality scores and how each dimension in the questionnaire relates to particular aspects of job performance.

What are 'Level A' and 'Level B'?

Level A and Level B are the two levels of training which form the principal stages in the British Psychological Society's (BPS's) certification system for psychometric training. Level A is the first stage and covers the basic principles of psychometrics and qualifies you to use and purchase ability and reasoning tests and other simple instruments. Level B is the second stage and qualifies you to administer and interpret personality questionnaires.

Level A and Level B courses are offered by the major test publishers and by other training providers in the UK. To offer these courses, an organisation must have been 'verified' by the BPS to ensure that those responsible for training are full conversant with the basic syllabus set out by the BPS and are assessing course participants appropriately.

What are the advantages and disadvantages of on-line testing as compared with the traditional pencil-and-paper method?

The principal advantages of on-line testing are convenience, speed and cost. With on-line testing, you do not need people to come to your premises to take a test. All you need is to send them the link and their entry codes (e.g. organisation code, access code, password) for your on-line facility and this allows them to take the tests and questionnaires on-line whenever and wherever is convenient to them.

As regards speed, as soon as the questionnaire responses are submitted, the data is analyzed and a report is created and e-mailed to you (with perhaps also a feedback report sent to the test-taker as well). All of this happens in less than a second after the data has been submitted.


16/18

As regards cost, the costs of on-line test administration and reports are very significantly cheaper than the cost of pencil-and-paper test materials and bureau-service reports. However, by far the greatest saving is in terms of staff time that would otherwise be involved in the test administration, scoring and report writing processes. With advanced online systems, there are also considerable advantages in terms of the administration and management of testing sessions like Genesys3.

A disadvantage of on-line testing is that it is not possible to make reliable assessments of ability in selection situations if the candidate takes the test remotely. Obviously, there would be nothing to prevent them having a more able friend sitting next to them telling them the answers to the question. This is not however to say that on-line testing cannot be used for ability assessment. Many organizations conduct on-line assessment of ability by getting candidates to take the test under supervised conditions while attending for interview. Although this requires the time of an administrator to supervise the testing, it still saves considerable time that would otherwise have been required for scoring and report writing

How are tests scored?

Tests are normally scored using an answer key of some sort. In the case of ability tests, the answer key provides the 'correct' answer to each question and usually a positive number (most commonly 1) is assigned for a correct answer and a zero is assigned for an incorrect answer (though there might be exceptions to this general rule). The scores are then totaled to obtain what is called the "Raw Score" on the test.

In the case of personality questionnaires, there are no right answers as such. In this case, particular responses will contribute to one or other side of a personality dimension. For example, suppose a personality questionnaire contains a question such as "I like going to parties", to which the possible responses are (a) agree, (b) uncertain and (c) disagree. If the person gives response (a), this might then contribute a score of 2 points to a scale of Sociability. A response of (b) might contribute just one point to this same scale and a response of (c) would contribute zero points. In the case of other questionnaires, the different responses to a given question might actually contribute to different scales. Whichever way it is done, one finishes up with a set of raw scores, one raw score for each dimension which is assessed by the questionnaire.

Except for certain special types of test, the raw scores by themselves do not actually provide very much information on how well the person has done or what their personality is like. Suppose for instance that a person scores 27 points out of a maximum of 50 on a test of verbal ability. On the face of it, that doesn't sound particularly good. However, if most other people who take the same test score below 20, then a score of 27 is extremely good. The same applies with the raw scores on personality dimensions. Is a raw score of 18 points out of 26 on a scale of Extraversion high or low? We don't know until we can find out how other people have scored.

For this reason, we normally take the scoring one final step further, using what are called 'norm tables'. Norm tables are used to convert raw scores into standardized scores. The standardized scores express the person's performance on the test in terms of a comparison with how a large sample of other people has performed. For example, using a norm table in the first example above, we might find that raw scores higher than 27 were obtained by only 5 percent of a large sample of people who took the test on a previous occasion (and conversely that 95 percent of people scored less or equal to 27). So the raw score of 27 is actually a very high score. We would express the result by saying that the person scored "at the 95th percentile".

Using the norm table is very simple. We just look up the raw score in one column and then go over to the percentile score in another column. Of course it goes without saying that if we are using a computerized or on-line test, all of this, including the basic


17/18

calculation of the raw scores is done for us by a computer program, so saving a great deal of administrative time.

As it happens, there are different types of standard score used with different tests. The percentile score as described above is the most common. But you may also encounter the Sten score (ranging from 1 to 10 with scores of 5 and 6 being in the middle), the Stanine score (ranging from 1 to 9 with a score of 5 at the middle) and the T-score (ranging from 20 to 80 with 50 in the middle).

Why are some tests timed and others not?

Generally speaking, personality questionnaires and other instruments which assess attitudes, interests and similar traits are not timed. The person can take as long as they wish to answer all the questions. Nevertheless, when administering a personality questionnaire, the test administrator may encourage the person taking the test not to spend too long thinking about the questions and to try to give the answer that comes to them most naturally and instinctively.

On the other hand, most tests of ability and aptitude are timed. It is important that the timing is very precise - just another minute working on the test might significantly improve a person's score. It is important therefore that candidates work as quickly as they can, though not so quickly that they begin to make mistakes. If a candidate gets bogged down trying to find the answer to a particular question, he/she may waste a great deal of time which could have been devoted to answering other questions. Normally it is possible to return to earlier questions so the best policy in such circumstances would be to move on to other questions and then return to the difficult question at the end.

How are psychometric tests constructed?

A full answer to this question would require a large technical volume. However, the basic principles can be set out simply.

The first stage in developing a test is to undertake research into the area of ability or personality that is to be assessed by the test. In the case of ability tests, this would involve researching the components that make up the ability in question, the way the ability is expressed in actual performance at tasks, the sorts of questions or exercises which would be tap into that ability and so on. In the case of personality it is somewhat more complex and involves looking into the nature of personality and how it is structured in terms of particular dimensions. Personality questionnaires are normally based on a particular 'model' of personality and it should be said that there a variety of different models of personality, each of which offers a different perspective upon human behaviour.

Once the initial research stage has been completed, the next stage involves writing trial test items (questions) and testing these out on preliminary samples of people. The purpose of this testing stage is to see if the items work in practice, whether they are reliable and whether each item appears to be assessing the ability or dimension of personality it is supposed to assess. This stage of the process can be very lengthy, involving several cycles of writing and rewriting test items and testing them out in practice.

The next stage is to assemble the best items from the initial trialing stages into something that will correspond to the final test. This test is once again trialed and the data from the trialing will allow a final version to be constructed.

The last stage is called 'standardization'. This is where the final version of the test is administered to a large sample of people (the 'standardization sample'), which is representative of the sort of people the test will be used on in the future. The sample


18/18

would include at least several hundred people and often significantly more that this. The data acquired during this stage is used to construct the norm tables which will later be used to convert the raw scores obtained from the test into 'standardized scores'. These standardized scores express an individual's test result in terms of a comparison with the performance of the people in the standardization sample.

Can tests be practiced?

There are competing arguments about whether practice affects performance on ability or aptitude tests. For tests such as the Scholastic Assessment Test (SAT), which is used to predict college performance in America, test preparation or ‘coaching’ is a big industry and many claims are made by coaching companies for the efficacy of their services.

The true claims for practice can be better understood if we look at two separate components that go to make up test performance. The first of these, and the one which test users are really interested in measuring, is a person’s true ability in the area being measured. This ability is likely to remain fairly static, although may fluctuate slightly with experience. For example, ability with numbers may increase if a person has to do a lot of number work and this, in turn, may be reflected in their score on a numerical reasoning test. Conversely, not using numbers for some time is likely to make a person far slower when completing the same numerical test, and so have the effect of lowering their overall score. Some improvements in score may occur if a respondent has recently taken the same test. In this case, improvement may be partially due to them remembering some of the test items and so working out the answers to them more quickly than the first time.

Despite this, coaching for actual test questions is likely to have very little effect on overall performance. This is because of the wide range of tests and item types available. A more effective way of preparing people, and one which is increasingly recognized as being best practice in psychometric testing, is to increase their familiarity with testing and so make them more ‘test wise’.

Many people use psychometrics because it allows them to compare respondents’ abilities on a ‘level playing field’. Companies may be faced with applicants with a range of experience and educational backgrounds, and this makes it difficult to make fair comparisons between them. With an ability or aptitude test, all respondents face the same set of questions under the same conditions. Whilst this increases the fairness of the comparison process, it needs to be recognized that respondents will have different levels of familiarity with testing and different levels of comfort with psychometrics. This familiarity or test wiseness can have a significant effect on performance.

Example and practice questions, which are common in psychometric tests, are designed to make sure that all respondents are equally familiar with the test. However, many tests also come with test preparation leaflets or ‘test taker’s guides’ which can be sent to respondents before they attend the testing session. The purpose of these is to help test takers understand what they will be asked to do, give them tips on how to prepare and give them examples of the types of question they will see in the test. Use of such guides is now recognized as best practice and, in some cases, full practice tests are used to give respondents detailed feedback in order to help them develop their test taking style.

why use psychometrics for employee selectionjp

Documents