Changes in selection to the Foundation ProgrammeProfessor Paul O’Neill

Chair, ISFP Project GroupMember UKFPO Rules GroupLead for Research and Evaluation Selection

Plan for Talk• Background to change – robust & numbers• Evidence around selection• SJTs

• Evidence • Piloting

• EPM• Algorithm• Academic FP

Selection (appointment)

Best prediction of the right person to do the best job

Should be done fairly

PON experience:• House jobs (Foundation)• CVs. application forms and letters• References & ‘putting a word in’• Interview panels – numbers, questions• Presentations• Occupational pyschologists• Personality testing

Now:• Mostly national• Application - anonymous, standardised and

assessed against criteria• Use of standardised tests (e.g. SJT)• Selection centres – standardised, multimodal,

some competency testing• Interview panels more & more standardised

Evolution2009 DH commissioned a review of selection

methods. The Improving Selection to the Foundation Programme project was set up and overseen by the Medical Schools

Council

2010/2011 New selection methods were piloted successfully

FP 2012Full-scale Parallel Recruitment Exercise (PRE)

FP 2013New selection methods implemented

A collaborative venture between

Educational Supervisors

Scottish Foundation BoardFoundation School Directors

Selection in MedicineUndergraduate

• Single School• UKCAT

Postgraduate• Foundation• Postgraduate – Birmingham Review

Substantive – Consultant & GP

Literature Reviews3 commissioned

WarwickNewcastleDurham

OverlappingSpecify methods usedIdentify gaps and where judgement will be needed

Durham: McLachlan & Turnbull• 236 References

• Behaviour predictive of future behaviour• Conscientiousness significant component of

concerns

Warwick: Thistlethwaite et al• 197 papers 1990-2009, medical school and

residency. • Most from USA

• Mostly looked at face validity and reliability NOT predictive validity

• Lack of consensus

• Non-medical literature – focus on employability skills (job analysis)

Newcastle: Illing et al• 202 references

• Foundation programme – narrow range of applicants, very few cannot do the job

• Selection methods must satisfy their stakeholders (employers, students etc)

• Cognitive tests are moderately predictive of later cognitive tests

• Non-cognitive elements need to be considered to ensure that a doctor is able to perform well

• Assessment centres allow for a range of methods to be used

Selection MethodsCombination of:

• Cognitive ability (academic)• Non-cognitive

Academic range very narrow• High ‘justice’

Incorporate measures of big 5 personality applied to medicine• Not IQ or general aptitude tests

Selection MethodsBig 5 Model of Personality is predictive of job and

academic performance• Extraversion (outgoing, sociable, impulsive)• Emotional stability (calm, relaxed)• Agreeableness (trusting, co-operative, helpful)• Conscientiousness (hardworking, dutiful,

organised)• Openess to experience (artistic, cultured, creative)

Assessment CriteriaReliability

ValidityConsistencyGranularity

Longevity

Will the process pick the ‘best’ applicants

across the UK?

Will the process be good for many

years?

Assessment Criteria

Will the process support or undermine educational

objectives?Educational Impact

How onerous is the process for applicants ?

Applicant Burden

To what extent does the process distract from

service delivery?

Clinician Time

Assessment CriteriaWill the process pick

minimise cheating or malpractice?

Compliance

Can the applicants see exactly where the

goal posts are?Transparency

Is there a level playing field?

Fairness (Justice)

Public Opinion Person in the street thinks that it is fair

Summary of Results

Better score

Lower 5-year cost

Selection to FP 2013Situational Judgement Test (SJT)

• SJTs will replace the ‘white space’ application form questions • This is an invigilated, machine markable test in exam conditions • The SJT will consist of 70 questions in 2 hours 20 mins

Educational Performance Measure (EPM)• The EPM will replace the academic quartile scores. The EPM score is

comprised of three elements: 1. medical school performance in deciles2. additional degrees3. academic achievements

The EPM and SJT will each be worth 50 points from a 100 point application score

The case for change

• ‘White space’ questions not sustainable as a selection tool• Will become steadily less discriminatory between eligible applicants (limited range of new questions that can be generated)

• SJTs draw upon bank of items to be available for each application round •Situations experienced in the Foundation Programme varied and complex• New items built incrementally and continuously against Job Analysis

• ‘White space’ questions non-invigilated conditions , model answers concerns about risk of plagiarism and coaching

• SJTs in invigilated conditions in the UK (3 national dates)• Not possible to revise for the SJT (scenarios complex, answers relate to judgement rather than knowledge)

Concern SJT and EPM

The case for change• Long-term technical reliability and validity could be improved

• 30 year evidence for reliability of SJTs • SJT pilots demonstrate the technical reliability, internal reliability, and validity for use for FP selection

•Academic quartile system - difficult to compare fairly between applicants from different medical schools (not standardised or subject to quality assurance across medical schools)

• EPM - standardised framework for deciles• Medical schools and students decide ‘basket of assessments’ • Schools will be required to publish their locally agreed deciles framework, which will facilitate transparency and quality assurance from the wider community. • Deciles fairer to applicants at margins

Concern SJT and EPM

Situational Judgement Test

What is a Situational Judgement Test?SJTs are:

• a test of aptitude• designed to assess the professional attributes expected of a

Foundation doctor• based on a detailed job analysis of an FY1 doctor

SJT questions assess your judgement by presenting you withchallenging situations you are likely encounter at work during the first year of an integrated Foundation Programme

Example SJT QuestionsThere are two question formats: 

• Rank the five responses in the most appropriate order• Choose the three most appropriate responses from eight

 You should answer what you ‘should’ do in the scenariodescribed, not what you ‘would’ do

Example Question 1 – rankingMr Reese has end-stage respiratory failure and needs continuous oxygen therapy.While you are taking an arterial blood gas sample, he confides in you that he knows heis dying and he really wants to die at home. He has not told anyone else about this as he thinks it will upset his family, and the nursing staff who are looking after him so well.

 Rank in order the following actions in response to this situation (1= most appropriate;5= least appropriate).

A. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospitalB. Reassure Mr Reese that the team will take account of his wishesC. Discuss his case with the multi-disciplinary team*D. Discuss with Mr Reese's family his wish to die at homeE. Discuss Mr Reese's home circumstances with his General Practitioner

Answer to Question 1B. Reassure Mr Reese that the team will take account of his wishesC. Discuss his case with the multi-disciplinary team*E. Discuss Mr Reese's home circumstances with his General PractitionerD. Discuss with Mr Reese's family his wish to die at homeA. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital

This question is focusing on effective communication with patients. Ensuring that patients’ informed wishes are met in relation to their care is central to your approach to patient care and this needs to be communicated to the patient in a reassuring manner even in situations relating to end of life care (B). These wishes should have been sought when addressing the management plan for Mr Reese and once identified the multidisciplinary team needs to be made aware of them in order to ensure that as far as possible Mr Reese’s views in relation to his end of life care are implemented (C). The management of Mr Reese will require the active involvement of his GP and communication with the GP is therefore of importance (E). Any decision to discuss Mr Reese’s wishes in relation to his end of life care with his family can only be made with the full agreement of Mr Reese (D). It would not be appropriate to give the patient inaccurate information in order to engineer a different medical pathway (A).

SJTs (Literature Review 77 papers)• Management, university, police, engineers• Large scale selection – short-listing• Construct validity not clearly identified

• Single construct (e.g. ‘practical intelligence’)• Can be designed to measure differing constructs

• Predictive validity will depend on what criterion is targeted

• SJT designed to test interpersonal skills will more likely predict inter-personal orientated performance

SJTs

• Used nationally to select GP registrars and other ‘high stakes’ occupations

• significant validity in predicting job performance • incremental validity over methods such as ability tests

and personality questionnaires • typically relate to general experience and ability, rather

than job-specific knowledge or experience • tend to show smaller differences between candidate

groups (e.g. based on race) than cognitive ability tests

Job analysis of FY1 doctor• Commitment to professionalism• Coping with pressure• Effective communication• Learning and professional development• Organisation and planning• Patient focus• Problem solving and decision-making• Self-awareness and insight• Working effectively as part of a team

Job analysis of FY1 doctor

Item development

Review, concordance,

piloting

Previously piloted & refined

(89 items)

Item writing workshops(43 items)

CIT interviews (78 items)

You have been prescribed codeine for persistent back pain which has become worse in the last few weeks. You have noticed that during shifts you are becoming increasingly tired, finding it difficult to concentrate and your performance, as a result, has been less effective.  Choose the THREE most appropriate actions to take in this situation  

Example Question 2 – multiple choice

A. Ask a colleague to assist with your workload until you finish your codeine prescription

B. Make an effort to increase the number of breaks during your next shiftC. Stop taking the codeine immediatelyD. Make an appointment to see your General PractitionerE. Seek advice from a specialist consultant about your back pain F. Arrange to speak with your specialty trainee (registrar)* before your next shift and

make them aware of your situationG. Seek advice from your clinical supervisor* regarding further support H. Consider taking some annual leave

Answer to Question 2D. Make an appointment to see your General PractitionerF. Arrange to speak with your specialty trainee (registrar) before your next shift

and make them aware of the situationG. Seek advice from your clinical supervisor regarding further support

This question looks at how you demonstrate commitment to professionalism and self-awareness. The essential problem is that as an FY1 doctor the level of your clinical performance is dropping. This constitutes a risk to the patients you are caring for and will impose a greater workload on your colleagues. In this circumstance you should inform and seek the advice of the senior clinician responsible for your work (G) and alert your colleagues (F). This matter is most likely to be related to your prescribed medicine and you should therefore consult with your GP (D) rather than any other specialist (E). It is not your place to re-allocate workload (A). Increasing the number of breaks is unlikely to improve a situation that is likely to be due to an adverse effect of a drug (B). You should not make any unilateral decisions about your medical treatment (C) and should seek the advice of others (D). You should not be seeking to use your annual leave (H) to compensate for a medical problem.

Scoring of the SJT Scoring key, determined through:• Consensus at item review stage (item writers, SMEs)• Expert judgement in concordance panel review• Review and analysis of the pilot data

Scoring not “all or nothing”, but based on how close to scoring key

Scoring of the SJT – ranking

• Up to 20 marks available

• Up to 4 marks available for each response (points for “near

misses”)

• No negative marking

Scoring of the SJT – multiple choice

• Points for each correct answer

• No negative marking

• 4 points for each correct answer

• Up to 12 points per item

A B C D E F G H

0 4 0 0 0 0 4 4

Parallel Recruitment Exercise (PRE)• New selection methods trialled alongside the normal selection

methods during 2012 FP application round

• Aims: logistics, awareness, pilot new SJT content

• All 31 medical schools involved

• SJT – 1 hr, 30 questions

• EPM – each medical school consulted study body on ‘basket of assessments’ to be used

Parallel Recruitment Exercise (PRE)• 90+% of FP applicants participated in the SJT

• Valuable learning experience ahead of live implementation

• Feedback to inform live implementation:

• Applicants

• Medical schools

• Was the final step in ensuring the selection methods can be consistently and robustly applied for implementation for FP2013

PRE - SJT30 item, one hour SJT

6,842 medical students took part in the PRE Participants included:

• Final year medical students• Students who had been pre-allocated to the Defence Deanery• Students who had chosen to take a year out post-graduate• International students returning overseas after graduation

30 medical schools (plus two centres for Eligibility Office applicants) delivered the SJT in 72 venues

Psychometric analysis shows that a 60 item SJT is a reliable measurement

Sheffield SJT pilot

Descriptive StatisticsInternal Reliability:

• Adjusted for a 60 - item test that included only robust items (such as would be used in an operational paper), all papers had an estimated reliability of α = 0.80 or above (α = 0.80 to α = 0.87)

• Demonstrates that the SJT is a reliable test in this context with items testing different things

Histograms showing the distribution of scores for Paper One (left) and Paper Two (right)

Descriptive StatisticsMean:

• Overall Mean = 79.5%; Range from 78.0% to 80.6%.So not too easy – differentiating appropriately

Standard deviations:• Mean SD = 18.6; Range from 17.3 to 20.0.

Distributions:• 305 to 468 (out of a maximum of 512) – as expected

given length of paper• Appears to be slightly negatively skewed, (more people

towards top end) although results do show a close to normal distribution

Item Facility (difficulty)Ranked Items

Maximum score 20Score 18 = ‘very easy’; Score 11.6 = ‘very hard’

Mean facility similar across all papers (approx 16)Range of facility values differed across papers

SD range similar for all papers, except for Paper 1 where one item

had a very high SD

Multiple Choice ItemsMaximum score 12

Score 10.8 = ‘very easy’;Score 3.6 = ‘very hard’

Mean facility across papers ranged from

7.9-9.0Range of facility values differed across papers

SD range was similar for all papers

Student views

The content of the assessment seemed relevant to the Foundation Programme

The scenario content seemed appropriate for my training level

Educational Performance Measure

Why change to EPM? • A clear framework with agreed principles used to calculate the EPM,

ensuring that it is fair, transparent and consistent across the schools of the UK

• Splitting cohorts into deciles rather than quartiles provides a wider spread of scores, which makes it easier to differentiate between applicants, and will be more fair for applicants at the margins

• It makes more sense for all the academic components of the application to be one part of the application

How is the EPM calculated? Score produced by applicant’s medical school to reflect achievementand performance compared to rest of cohort

EPM = 3 parts (maximum 50 points):1. Medical school performance in deciles (34 – 43 points)

E.g. Top 10% = 43; Top 20% = 42; etc2. Additional degrees (max 5 points)3. Educational achievements (presentations, prizes and

publications (max 2 points)

Schools have consulted with students about how the decile points for the EPM will be calculated.

PRE - EPM• 27 of 30 medical school initiated a new consultation and review of

framework• (3 schools consult annually and framework aligns)• Other schools do consult annually – but undertook new consultation with

students for the PRE• Majority of schools pleased with student engagement, especially amongst

later years• Benefit of raising awareness with students & staff• Students felt sense of ownership

Meetin

g stud

ent re

ps

Committe

e

Email to

all stu

dents

Online s

urvey

Open m

eetin

g

Studen

t reps

cons

ulted

Conve

ned w

orking

grou

p0

2

4

6

8

10

12

14

16

18

20

Consultation on frameworks

PRE EPM – Decile Points Decile/ Quartile

4th 3rd 2nd 1st

1st 2 6042nd 1 3 18 6463rd 1 5 341 3344th 2 13 649 275th 10 52 606 86th 13 625 50 27th 37 629 19 38th 337 348 109th 680 30 810th 671 7 2

PRE EPM – successes• All medical schools have agreed a ‘basket of assessments’ in

consultation with students

• All medical schools aligned with EPM framework

• All medical schools calculated EPM deciles, with around 10% in each decile (some ties)

• All medical schools confident they can calculate EPM deciles in line with common principles

Academic Foundation Programme2012 Same timetable as for standard applicationApply to 2 UoAHave to sit SJT

Appointed – application, EPM, interview (+/-)

If not appointed, then revert to standard process

Algorithms• Was initially triggered by student preferences

• Unstable, unfairness• Worse if increasing number applicants/school

• Changed for FP2012• Now triggered by application score

• Has changed patterns of applications

Selection to the Foundation Programme – improving and evolving

FP 2005Foundation Programme introduced

FP 2006National timetable and application process

FP 2007 Online application – white space & quartiles

2009-2011 Improving Selection to Foundation Programme

FP 2012Full-scale Parallel Recruitment Exercise (PRE)

FP 2013New selection methods implemented

‘The greatest forward step in the baking industry since bread was wrapped –

Missouri, 1928

What’s Wrong with SJTsLawrence Clinical Pharmacology

Wonderful

Not fit for my dog

Useful in some situations

More informationUKFPO - www.foundationprogramme.nhs.uk

• FP 2013 Applicant Handbook• Introductory videos• SJT monograph• SJT practice paper• FAQs (FP 2013, SJT, EPM)

Archived ISFP website – www.isfp.org.uk

Any questions?