GP and Specialty SelectionGP and Specialty Selection

Professor Fiona Patterson Professor Fiona Patterson Work Psychology Group & University of CambridgeWork Psychology Group & University of Cambridge

Professor Jane DacreProfessor Jane DacreUniversity College LondonUniversity College London

OverviewOverview

Principles of best practice selectionPrinciples of best practice selection Defining selection criteria & the criterion problemDefining selection criteria & the criterion problem Accuracy of different selection methodsAccuracy of different selection methods

Overview of selection methodsOverview of selection methods Overview of selection methodsOverview of selection methods Situational Judgement TestsSituational Judgement Tests

GP & Specialty SelectionGP & Specialty Selection

Selection centresSelection centres

AoMRCAoMRC Clinical problemClinical problem--solving test pilotsolving test pilot

Designing selection systemsDesigning selection systems “…“…the function of selection is to provide a means of estimating the function of selection is to provide a means of estimating

candidates’ aptitude so that the most suitable applicants are candidates’ aptitude so that the most suitable applicants are appointed…appointed…”.”.

Selection uses a Selection uses a predictivistpredictivist paradigm. In exams, end of paradigm. In exams, end of training capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can pass

Selection aims to Selection aims to predict trainabilitypredict trainability, i.e. identify those who will , i.e. identify those who will successfully complete training, successfully complete training, beforebefore it commences. it commences.

Parameters for Parameters for designing & validating designing & validating selection systems are selection systems are different to other assessment settingsdifferent to other assessment settings

Create competency model

Create person specification

Knowledge, skills, abilities,

characteristics

JobAnalysis

Tasks, roles, responsibilities

Competency analysis

Choose selection methods

Interviews, Simulations,

Selection centre

Attract pool of applicants

RecruitmentSelect out unsuitable applicants

The Selection ProcessThe Selection Process

Identify selection criteria

Make selection decisions

characteristicsRequired level of

competence

Selection centre

Reliability, validity, financial utility,

fairness

Validate selection decisions

SelectionSelect in and

rank

Aptitude?Attainment?

Patterson & Ferguson, 2007. Patterson & Ferguson, 2007.

What attributes are important to be an effective clinician?

How are they best tested at selection?How are they best tested at selection?

Beyond some basic assessment – is a lottery the best option?

Selection methodSelection method Evidence Evidence for Validityfor Validity

Applicant Applicant ReactionsReactions

Extent of Extent of UseUse

Structured interviewsStructured interviews HighHigh Moderate to positiveModerate to positive HighHigh

Cognitive abilityCognitive ability HighHigh Negative to moderateNegative to moderate ModerateModerate

Situational Judgement TestsSituational Judgement Tests HighHigh ModerateModerate ModerateModerate

Personality testsPersonality tests ModerateModerate Negative to moderateNegative to moderate ModerateModerate

BiodataBiodata HighHigh ModerateModerate ModerateModerate

Work sample testsWork sample tests HighHigh PositivePositive LowLow

Assessment centresAssessment centres HighHigh PositivePositive Moderate Moderate

GraphologyGraphology LowLow Negative to moderateNegative to moderate LowLow

ReferencesReferences LowLow PositivePositive HighHigh

LotteryLottery N/AN/A Negative Negative LowLow

Innovations in selectionInnovations in selectionmethodsmethodsmethodsmethods

Situational Judgement TestsSituational Judgement Tests

What’s a Situational Judgement Test?What’s a Situational Judgement Test?

Situational Judgement Tests (SJTs) are a Situational Judgement Tests (SJTs) are a measurement measurement methodmethod designed to assess judgement in workdesigned to assess judgement in work--relevant relevant situations:situations:

Present challenging situations likely to be encountered at workPresent challenging situations likely to be encountered at work

Make judgements about possible responsesMake judgements about possible responses

Scored against preScored against pre--determined keydetermined key

Validity (predictive & incremental) is wellValidity (predictive & incremental) is well--establishedestablished

Recent examples in postgraduate specialty selectionRecent examples in postgraduate specialty selection GP, CMT, Surgery, Acute specialties, Public HealthGP, CMT, Surgery, Acute specialties, Public Health

Example SJT item

You are reviewing a routine drug chart for a patient with rheumatoid arthritis during an overnight shift. You notice that your consultant has inappropriately prescribed methotrexate 7.5mg daily instead of weekly.

Rank in order the following actions in response to this situation (1= Most appropriate; 5= Least appropriate).

A Ask the nurses if the consultant has made any other drug errors recently

B Correct the prescription to 7.5mg weeklyC Leave the prescription unchanged until the consultant ward round

the following morning D Phone the consultant at home to ask about changing the

prescription E Inform the patient of the error

Medical Education, 2009

‘Job families’ of SJTs

Job family

1. Interventionist Surgery, Radiology, O&G

2. Family/Community-based

SJT Pilot studies

Surgery SJT pilot (Rowley et al)

Radiology SJT evaluation (Lindsell et al)

GP SJT (Irish, Jones, GPNRO) 2. Family/Community-based GP, Medicine, Psychiatry, Paeds

3. Acute Anaethesia, Emergency

medicine

4. Laboratory-based Histopathology

CMT SJT evaluation (Berkin, Burr, Lazell et al)

Public Health pilot (Pashynan, Williams)

Acute specialties SJT pilots (Gale, Anderson, Carr et al)

GP selectionGP selectionGP National Recruitment OfficeGP National Recruitment OfficeGP National Recruitment OfficeGP National Recruitment Office

GP SelectionGP Selection

6,500 applicants per year

job analysis to develop selection criteria

longitudinal validation studies

analysis of applicant reactions analysis of applicant reactions

Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A. A competency model for general practice: implications for selection, training, and development. Br J Gen Pract 2000;50(452):188-93.

Patterson F, Ferguson E, Norfolk T, Lane P. A new selection system to recruit general practice registrars: preliminary findings from a validation study. BMJ 2005; 330(7493):711-4

National Deanery

GP Specialty Selection

Longlisting Shortlisting

Stage 2

Selection Centre

Stage 3

ElectronicApplicationProcess

GroupWritten SimulationClinical Problem solving testSituational Judgement test

Ranking

Matching to Deanery

Foundation Competency

National panel

GP Selection SystemGP Selection SystemPredictive & construct validity studies Predictive & construct validity studies Predictive & construct validity studies Predictive & construct validity studies Study 1. Predictive validity of short-listingStudy 2. Comparing the CPS, SJT & cognitive ability testsStudy 3. Supervisor ratings 1 year into trainingStudy 4. MRCGP outcomes after 2-3 years in training

Study 1. Predictive validity

CorrelationsCorrelations N N CorrelationsCorrelations

CPS & SJT 8399 .53**

CPS & Selection Centre 6824 .41**

SJT & Selection Centre 6824 .49**

Prediction Prediction of of SC SC (N=6824)(N=6824) AdjAdj RR22 RR2 2 changechange

CPS & SJT 0.28CPS & SJT 0.28

Incremental prediction of CPS over SJT .04**

Incremental prediction of SJT over CPS .11**

• SJT is the best single predictor for short-listing• Both tests are strong predictors & show incremental validity over the other

** p<.001 CPS – Clinical Problem-Solving test. SJT – Situational Judgement test

Study 2. Comparing the validity of CPS, SJT & cognitive ability tests

Study 2. Comparing the validity of Study 2. Comparing the validity of CPS, SJT & cognitive ability testsCPS, SJT & cognitive ability tests Although the cognitive ability tests were correlated with SC Although the cognitive ability tests were correlated with SC

outcomes, the current outcomes, the current shortlistingshortlisting tests remained the best tests remained the best predictors overall, particularly the SJTpredictors overall, particularly the SJT

Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, especially for relevance & opportunity to demonstrate skills especially for relevance & opportunity to demonstrate skills

CorrelationsCorrelations

N=196 N=196 MeanMean SDSD αα CPSCPS SJTSJT SCSC

Supervisor ratings after 12 months24 items

4.63 .86 .94 .48** .50** .47**

Study 3. Predictive validity in trainingStudy 3. Predictive validity in training

** p<.001

• All methods (CPS, SJT & SC) are highly significant predictors of in-training performance

• Significant incremental validity accounted for by the SC for important domains (e.g. communication skills, empathy)

• Results indicate priorities for improving efficiency

Study 4. LongStudy 4. Long--term Predictive validityterm Predictive validity

Mean CorrelationsMean Correlations

NN MeanMean SDSD CPSCPS SJTSJT SelectionSelection CentreCentre**

MRCGP Applied Knowledge Test 6 diets

2,550 150.6 96.6 .76** .47** .29**Knowledge Test 6 diets

MRCGP Clinical SkillsOSCE 5 diets

1,277 9.3 2.2 .35** .48** .35**

* Uncorrected correlations for restriction of range ** p<.001

• The GP selection methods are highly significant predictors of end-of-training competence

Speciality SelectionSpeciality Selection

Specialty Specific Definitions of Specialty Specific Definitions of Communication SkillsCommunication Skills

PaediatricsPaediatrics Communication SkillsCommunication Skills

Is able to switch conversational style from adult to child, and to manage Is able to switch conversational style from adult to child, and to manage the doctorthe doctor--patientpatient--parent triad during consultation; engages child in parent triad during consultation; engages child in conversation; positions self at child’s level when speaking to childconversation; positions self at child’s level when speaking to child

AnaestheticsAnaesthetics Communication SkillsCommunication Skills

Is able to Is able to verbaliseverbalise intentions, explain actions to various people in a intentions, explain actions to various people in a

Obstetrics & GynaecologyObstetrics & Gynaecology Communication SkillsCommunication Skills

Is able to discuss psychoIs able to discuss psycho--sexual problems with patient; uses open sexual problems with patient; uses open questions or puts ‘feeler’ questions out to patients who may be shy / questions or puts ‘feeler’ questions out to patients who may be shy / embarrassed to discuss symptoms; uses cues (such as fetal heart embarrassed to discuss symptoms; uses cues (such as fetal heart monitor) to illustrate explanationsmonitor) to illustrate explanations

Is able to Is able to verbaliseverbalise intentions, explain actions to various people in a intentions, explain actions to various people in a team (e.g. surgeons & ODAs); asks open questions at preteam (e.g. surgeons & ODAs); asks open questions at pre--med to med to ascertain the facts ascertain the facts -- rather than just running through a checklist; can rather than just running through a checklist; can offer clear explanation quickly to patients with establishing an onoffer clear explanation quickly to patients with establishing an on--going doctorgoing doctor--patient relationshippatient relationship

Job analysis to define selection criteria Job analysis to define selection criteria for Surgery for Surgery

Technical Knowledge Technical Knowledge & Clinical Expertise & Clinical Expertise

Judgement under Judgement under PressurePressure

Organisation & Organisation & PlanningPlanning

Personal AttributesPersonal Attributes Decision MakingDecision Making Professional Professional IntegrityIntegrityIntegrityIntegrity

Leadership & Team Leadership & Team InvolvementInvolvement

Situation Situation AwarenessAwareness

Learning & Learning & DevelopmentDevelopment

Communication SkillsCommunication Skills Problem SolvingProblem Solving Practical skillsPractical skills

Key messagesKey messages

1.1. Research evidence shows there is more commonality than Research evidence shows there is more commonality than difference between the specialtiesdifference between the specialtiesegeg. communication skills, professional integrity, problem. communication skills, professional integrity, problem--solvingsolving

2.2. Research shows different priorities between specialties at Research shows different priorities between specialties at 2.2. Research shows different priorities between specialties at Research shows different priorities between specialties at selectionselection

Paediatrics:Paediatrics: Communication skills, EmpathyCommunication skills, Empathy O&G: O&G: Professional Integrity, EmpathyProfessional Integrity, Empathy Anaesthesia:Anaesthesia: Vigilance & Situational awarenessVigilance & Situational awareness

AoMRCAoMRC MMT PilotMMT Pilot

Investigating:Investigating:

1.1. If a single test could work for a range of specialtiesIf a single test could work for a range of specialties

2.2. If several specialties & deaneries can collaborateIf several specialties & deaneries can collaborate

3.3. Which test format is best Which test format is best -- a CPS, an SJT or botha CPS, an SJT or both3.3. Which test format is best Which test format is best -- a CPS, an SJT or botha CPS, an SJT or both

4.4. If computer based testing makes sense with respect toIf computer based testing makes sense with respect to costcost logisticslogistics stakeholder acceptabilitystakeholder acceptability

AoMRC MMT Pilot

Multispecialty/multideanery

Computer based

CPS & Acute Specialties SJT

Anaesthetics, ACCS, GP, Anaesthetics, ACCS, GP, Histopathology, Medicine, & Paediatrics

25 locations, over 2 days

N= 682 approx

Selection centresSelection centres

Origins & evidenceOrigins & evidence When designed appropriately When designed appropriately SCs areSCs are the best predictor of the best predictor of

future work performance for future work performance for nonnon--cognitive skillscognitive skills RReliabilityeliability && validity validity gains gains -- combining different selection tools, combining different selection tools,

standardised scoring systems standardised scoring systems SCsSCs usesuses jobjob--relevant exercisesrelevant exercises,, observation by observation by independent, independent, SCsSCs usesuses jobjob--relevant exercisesrelevant exercises,, observation by observation by independent, independent,

trained assessors trained assessors Recent reviews estimate over 95% of large organisations Recent reviews estimate over 95% of large organisations

employing more than 10,000 individuals, use SCs for graduate employing more than 10,000 individuals, use SCs for graduate recruitment & almost all high stakes selection used SC recruitment & almost all high stakes selection used SC techniquestechniques

Acceptability &Acceptability &candidate reactionscandidate reactions

Candidate Reactions: Percentage Candidate Reactions: Percentage agreement reported for each methodagreement reported for each method

2008 2009

S Listing N=5866

Selection Centre N=4631

S ListingN=2894

Selection CentreN=3571

CPS SJT Sim Written Group CPS SJT Sim Written Group

Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76%

Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81%

Appropriate 78% 57% 91% 80% 83% 86% 69%

Opportunity 56% 38% 75% 47% 49% 73% 35%

Candidate Reactions: Percentage Candidate Reactions: Percentage agreement reported for each methodagreement reported for each method

2008 2009

S Listing N=5866

Selection Centre N=4631

S ListingN=2894

Selection CentreN=3571

CPS SJT Sim Written Group CPS SJT Sim Written Group

Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76%

Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81%

Appropriate 78% 57% 91% 80% 83% 86% 69%

Opportunity 56% 38% 75% 47% 49% 73% 35%

Patient simulation

viewed positively

Implications

• High fidelity assessments (valid, but costly) are rated as significantly more relevant & fair by candidates

• Introduced methods beyond academic qualifications to select doctors (eg. interpersonal skills, integrity, empathy)

• Disjoint between operational validity vs. stakeholder acceptance ie. - “Why shouldn’t climbing Everest count”?

• Political validity important concept in this context

SummarySummary

Undergraduate and specialty postgraduate Undergraduate and specialty postgraduate ‘selection’ may have different goals ‘selection’ may have different goals --undergraduate courses selecting primarily on undergraduate courses selecting primarily on academic ability, with a focus on passing the academic ability, with a focus on passing the course and postgraduate selection focuses more course and postgraduate selection focuses more course and postgraduate selection focuses more course and postgraduate selection focuses more on jobon job--fit…it cannot necessarily be assumed that fit…it cannot necessarily be assumed that those with high academic ability alone can be those with high academic ability alone can be turned into potentially good doctors via medical turned into potentially good doctors via medical training training -- other skills & qualities may need to be other skills & qualities may need to be present from the start.”present from the start.”

Thank youThank you