SelectionDevelopment Review

Volume 23, No. 1, 2007ISSN 0963-2638

&SelectionDevelopment Review

P U B L I S H E D B Y T H E B R I T I S H P S Y C H O L O G I C A L S O C I E T Y

TO MAKE THE MOST OF THE AVAILABLEselection technologies, those involved in staffassessment need to be bilingual; fluent in thelanguages of both competencies and psycho-metrics. Competency metrics provide a way ofmanaging this interface that is reliable,predictable and accountable. The term ‘compe-tency metrics’ refers to the alignment of person-ality measures with workplace competencyassessments using algorithmic techniques. In thiscontext, algorithms are mathematical operationsdesigned to capture the relationships betweenvarious combinations and weightings of person-ality scales on the one hand, and behavioursassociated with workplace competencies on theother. This has in the past usually been a matterof individual judgement and expertise ratherthan an accountable, systematic numeratelyliterate process. In this new approach, tradi-tional psychometrics are used to create a person-ality profile from an individual’s responses topersonality questionnaire items. Competencymetrics take you from personality profile tofinely incremented competency rating; in effect,telling you to what extent an individual’stemperament facilitates or interferes with eachassessed competence.

Psychometrics meet competenciesThe style and design of psychometric personalitytests has not changed significantly over manyyears. Whether an instrument reflects a particulartheory or model of personality, trait or type theoryor, like current FFM questionnaires, is based onempirical meta-analytic studies of data generatedacross all these various approaches, the basicpsychometric technology is pretty much the same.There is also a similarity in the nature of theoutput in profile form. Because the results ofthese questionnaires require professional inter-pretation, psychometric testing also supports anextensive training industry. Like pharmaceuticalproducts, personality profiles are deemed unsafein the hands of the un-initiated. It seems that HRprofessionals have been summoned to the moun-tain of psychometrics, rather than the mountainof psychometrics going to the Mohammed of HR.

Even the seismic impact of the internet hasmade surprisingly little difference so far topsychometric testing. Most online assessmentsystems are little more than paper-and-pencilquestionnaires presented on website pages. Themost dramatic influence of the internet onassessment seems to have been in expanding thepsychometric using population, an acceleratingtrend because of rapidly increasing transmissionspeeds and accessibility. The factor that has hada powerful influence on the nature of candidateand employee assessment is the almost universaladoption of competency frameworks fordescribing the behaviours necessary for effectivejob performance.

Addressing the variability of profileinterpretationThe language of competencies now provides thepredominant vocabulary for describing jobrequirements as well as the basis for candidateassessment practices such as assessment centresand competency based interviews and appraisals.Not surprisingly, there has also been pressure forthe results of psychometric assessments to beexpressed in the same terms. At PCL, as at otherconsultancies I am sure, for many years we havebeen preparing individually crafted reports inwhich personality assessment has been shoe-horned into the client’s competency framework.This labour intensive approach to the prepara-tion of what we described as ‘Profile Match’reports had two serious shortcomings. Firstly,they were time consuming and expensive tocreate. Secondly, it proved extremely difficult tocontrol inconsistencies in personality profileinterpretation and the resulting competencyratings across candidates. However, this era ofhand crafting Profile Match reports gave us aproving ground within which this earlier expertjudgement based process became more rigorousand systematic. The need to address these diffi-

Selection & Development Review, Vol. 23, No. 1, 2007 3

SDRCompetency Metrics

Geoff TrickeyPCL

4 Selection & Development Review, Vol. 23, No. 1, 2007

culties provided the driving force behind thedevelopment of competency metrics and thecurrent automated online system –PROFILE:MATCH® – which was launched lastyear.

Guiding principles1. That only five factors of personality can be

independently measured. Beyond this, scalesare either elements within a factor (and willtherefore overlap with other similarelements), or they address content fromoutside the personality domain (e.g. occupa-tion interest, motivation or values items. SeeHofstee, 1990).

2. That the five personality factors can accountfor the extensive range of workplace behav-iours – in the same way that just three primarycolours provide more tones and hues than canbe distinguished by the human eye (Wiggins,1979; Mount & Barrick 1995).

3. That to reflect the diversity of personality aswe experience it in daily life, these five‘primary colours of personality’ need to becombined – rather than fragmented intoinnumerable scales.

4. Fragments of personality scales (HICs, sub-themes, etc.) are inherently less stable thancomplete scales and inferences cannot bedrawn from them with the same degree ofconfidence.

5. A temperament-based competency rating maydraw from any permutation of the five person-ality factors. Tests that pursue the fine detailof personality by proliferating scales, in ourview, seem to miss this important point.

6. Overlaying or blending complete scales invarious ways to achieve the fine detail ofpersonality (including alignment of person-ality with competencies) builds numbers ofcontributing items, and achieves more robustand stable measurement.

The processWe identify six steps in the process of aligningpersonality with competencies, although devel-opment will involve cycling back and forththrough earlier steps to make refinements. Stepsone and two below achieve a first approximationof personality alignment to the competency util-ising the same knowledge of and experiencewith the dimensions of personality as inform theusual processes of profile interpretation. Clearly,our emphasis is on personality and this means

that the relationship with a competency definedin terms of skills, knowledge and experiencemust be partial. In order to address this, thecompetency definition has to be revised so that itis clear what is and is not being measured. Thefirst three steps below are inferential, the finalthree steps cover the competency metrics. 1. The personality characteristics likely to

contribute to a competency are identified(e.g. a definition of creativity is likely toinclude curiosity, imagination and readinessto challenge the status quo).

2. The nature of the relationship of each person-ality characteristic to the competency isconsidered (e.g. whether linear, curvilinear,and whether positive or negative).

3. The definition of each competency is revisedand finalised to reflect this personalitycontent.

The remaining steps are concerned with trans-lating these judgements into mathematical algorithms:4. The personality/competency relationships

are expressed as algorithms that will definetheir contribution to a competency rating.

5. The appropriate weighting of each personalityscale contributing to a competency rating isestimated.

6. The composite of the contributing algorithmscan then be run against a 1000+ database ofpersonality data to evaluate distribution char-acteristics of the resulting competency ratings.

Summarising the benefitsThe development of competency metrics beganas an in-house project addressing the issues weconfronted in our efforts to meet client requestsfor assessments aligned to their competencyframeworks. What began as a PCL utility hasended up as an innovative resource for otherassessment practitioners. Having initially createdpersonality assessment and competency-basedreport generating systems for internal PCL use,it became evident that others could operate thisutility just as well as we could.

The project more than met its original objec-tives. The intended benefits were realised and, inaddition, we became aware of a number of unin-tended and positive consequences of the compe-tency metric approach as we developed theonline system.

Expected benefits of competency metrics:1. They reflect the wide adoption of competency

frameworks to describe and assess workplaceperformance and fit well alongside assessmentcentres and competency-based interviews,appraisals and other organisational perfor-mance measures.

2. They allow assessment to be given a specificfocus from the outset, addressing relevant andspecific questions, rather than commissioningthe same full personality profile for a varietyof different roles.

3. They put the interpretation of personalitydata onto the same explicit and accountablefooting as the tests that generate the person-ality profile. They can be researched and vali-dated.

4. They guarantee that all candidates are treatedin the same way.

5. Algorithms can be refined and developed toimprove the measurement qualities, distribu-tion characteristics and predictiveness ofcompetency ratings, on the basis of experi-ence and research.

Unintended consequences:6. Psychometric methods emphasise correlation-

based techniques such as regression, factoranalysis and principal components analysis.Non-linear relationships are invisible to corre-lation, yet may be crucial to performancepredictions. Competency metrics readilyaccommodate such relationships.

7. Because competency metrics combine scales,rather than fragmenting them – inferencesabout performance are: (a) based on largenumbers of items; and (b) are able to drawfrom the wide knowledge base surroundingthe FFM primary scales.

8. From the user perspective, the online systemoperates entirely within the user friendlylanguage of competencies. With the currentsystem, users only need to know what they arelooking for in competency terms. The psycho-metrics operate entirely ‘behind the scenes’.

9. While competency ratings used in assessmentcentres, competency based interviews orappraisals will typically be on a three-, five- oroccasionally a seven-point rating scale – ourcompetency metric technology is actuallybased on a scale of 45 (effectively the func-tioning range of the T scale), reduced – forpragmatic concerns about over-interpretation– to a 15-point scale.

DiscussionSince, in effect, the original hand-crafted ProfileMatch reports followed the typical routine ofdrawing inferences from personality data on thebasis of professional experience and generalisa-tions from applied research, our group reportwriting experience sensitised us to the vulnera-bility of standard personality interpretationprocesses. When you have several PCL psycho-logists simultaneously attempting to alignpersonality with competency frameworks on ajudgement basis, the consistency problemsbecome very apparent. How do you ensure thatthe emphasis and weight given to any one scalescore is sufficiently similar across all reportcompilers – even if very tight agreements aboutthe emphasis to be given for each scale in rela-tion to each competency have been established?The simple fact is that you cannot – and couldnot – even if all the reports were to be written bythe same individual. These experiences exposethe weakness of the established controls in theuse of personality assessments – the consistencyand reliability of the questionnaires may be opento scrutiny, criticism and review, but the inter-pretation of the profiles being generated at the pointof delivery is not. In other words, when usinganything other than the most literal andsimplistic personality questionnaires, onetrained and experienced test user may drawdifferent conclusions from a profile thananother. While there may be valiant efforts onthe part of the Society and CIPD to expose theweakness of any instrument (through reviews,training, ‘best practice’ initiatives, etc.), there isno obvious way of exposing the variability ofprofile interpretations based on professionaljudgement.

Competency metrics address this issue headon. This is a methodology that puts profile inter-pretation onto the same accountable basis as thetests themselves. Being mathematically tied tothe personality scales on which thePROFILE:MATCH® competency ratings arebased, the measurement quality comes throughthe algorithmic process. These ratings are in factmetrics based on a T-scale and they arecompletely objective and completely consistentacross candidates. They are also amenable tovalidation and research at a level of precisionthat individual profile interpretations are clearlynot. In a recent HMRC project, we used compe-tency metrics to align personality test scores tothe seven HMRC competency definitions.

Selection & Development Review, Vol. 23, No. 1, 2007 5

6 Selection & Development Review, Vol. 23, No. 1, 2007

The overall competency ratings generated bycompetency metrics for project participantscorrelated 0.46 with ratings achieved by aselected sample considered at HMRC to exem-plify the characteristics enshrined in theircompetency definitions.

The next stepNow that PROFILE:MATCH® is up and runningas an open access system, we are preparing aversion for use by those professionally trained inoccupational testing, PROFILE:MATCH PROTM.This will allow psychometric practitioners to usea simple user-friendly interface to develop theirown competency definitions and algorithms andadd their own new competencies to the compe-tency library, as it appears within their ownaccount area on the online system. They will, ineffect, go through each of the steps originallyinvolved in creating PROFILE:MATCH®, asdescribed above. PROFILE:MATCH PROTM willallow practitioners to align personality assess-ment with the competency frameworks of theirown clients with the assurance of completeconsistency of competency rating across allcandidates and all the other benefits of thecompetency metric approach.

ReferencesHofstee, W.K.B. (1990). The use of everyday

personality language for scientific purposes.European Journal of Personality, 4, 77–88.

Mount, M.K. & Barrick M.R. (1995). The BigFive personality dimension: Implications forresearch and practice in human resourcemanagement. Research in Human ResourceManagement, 13, 153–200.

Trickey, G. (2006). Linking personality data toHMCE’s competency framework. Strategic HRReview, 5(4).

Wiggins, J.S. (1979). A psychological taxonomyof trait-descriptive terms: the interpersonaldomain. Journal of Personality and SocialPsychology, 37, 395–412.

WWW.BPSBLACKWELL.CO.UK

The British Psychological Society and Blackwell Publishing’s unique partnership –

BPS Blackwell – delivers an essential range ofbooks for those seeking information on the latestpsychology practices, research and methodology.

As a BPS member, you are entitled to a 20% discount on any BPS Blackwell title.As a BPS student member, you are entitled to a

30% discount on any BPSBlackwell title.

Visit the website for further information www.bpsblackwell.co.uk