the royal college of pathologists – the overall assessment system dr. trevor gray director of...
Post on 29-Dec-2015
216 Views
Preview:
TRANSCRIPT
The Royal College of Pathologists – the overall
assessment system
Dr. Trevor GrayDirector of Examinations and
Assessment
The Assessment System
Overview of PMETB and assessment strategy
How WBA fits with other assessments Advantages and disadvantages of WBA
compared with other assessments Local organisation of WBA and how it is
administered by the College Future prospects
Overview PMETB (and the MMC process) has laid down
significant challenges to medical postgraduate training (elegantly dissected in the Tooke report)
PMETB principle is that training and assessment should be much more integrated – an assessment system
Workplace-based assessment is an essential element of the assessment of trainees undertaking specialist medical training
The specialty curriculum, training, exams and assessments are core College business (in the charter)
The College’s aim is to develop a validated assessment system which is fit for purpose
PMETB strategy on assessment
PMETB principles for assessment 2004 Nine principles for assessment systems Now backed up with technical documentation
Principles based on “overarching strategy” Workplace-based assessment Examinations of knowledge Examinations of Clinical Skills(“Workplace-based assessment” PMETB, Jan 2005)
Specialty curriculum and assessment system now approved by PMETB
(i.e. match strategy)
Workplace-based Assessment
Assessment based on performance in the workplace and measured there
Assessed locally by supervisors as well as staff of all grades in contact with the trainee
Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance
Results put into trainee portfolio for their annual review and hence important for trainee development and ultimately their employment contract
College Assessment Systemapproved by PMETB Workplace-based assessment Year 1 assessment MRCPath Part 1 (test of knowledge) MRCPath Part 2 (test of skills and applied
knowledge)
Feedback on regular basis (WBA) Educational supervisor’s report ARCP (Annual Review of Competency
Progression)
The College Assessment system: Miller’s pyramid
Knows
Knows how
Shows
Does
MRCPath Part 1
MRCPath Part 2 Practical
WBA e.g. Mini-CEX, DOPS
WBA - MSF
Value of multiple assessments Key is several
modes of assessment – triangulation
A single exam system (MRCPath) does not provide adequate coverage of clinical practice
WBA is also not sufficient in itself
(Assessment System Guidance – PMETB)
Examinations and Assessment
Examinations are simply one particular form of assessment
Assessments are: formative (involve feedback and are intended
to guide training) or summative (assess whether reach given
standard) e.g. driving test, MRCPath All assessments should be “blueprinted”
to the curriculum to demonstrate that the most appropriate method is being used
How Workplace-based Assessment fits with MRCPath
Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee
Examinations provide an independent assessment of knowledge and practical skills to national standards
Workplace-based assessment provides a local insight into candidates’ performance in routine tasks and interaction with staff and patients
How Workplace-based Assessment fits with MRCPath
Workplace-based assessment is a local process governed by deaneries albeit to PMETB standards
Examinations are organised nationally by the College
Comments, complaints and appeals for WBA go to deanery, for exams go to College
Workplace-based Assessment – Advantages
Measurement occurs in more authentic environment than the simulated one of the examination room
Trainees can take assessment in their own time and are not limited to exam timetable
The assessment tools are set up to provide extensive feedback to the candidate (formative)
Use of computerised forms makes them easy to record
Workplace-based Assessment
– Disadvantages To avoid time-consuming “busy”
assessment forms, the assessments are often simplified if not simplistic
A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties
They have only been validated in a number of clinical settings but not in all pathology settings
Computerised forms can over-simplify complex judgements and may not reward excellence appropriately
Year 1 Assessment (OSPE)
Developed to provide evidence of progression in run-through training in the absence of further recruitment and selection
Designed to assess progress in first year and be taken after 7 months training
It is a test to “indicate the capability and potential of a trainee through a test of applied knowledge and skills relevant to the year 1 curriculum”
F1 F2 ST1 ST2 ST3 ST4 ST5
WBA
Y1A
MRCPath
CCT Part 1 Part 2
MSF
Workplace-based assessment
ARCP
Foundation Stage A Stage B
Stage C
Stage D
Training and Assessment Plan
The training environment Workplace-based assessment utilises the
skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff
The College cannot train or quality assure all assessors
Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised and training is cascaded from those attending events such as this
Workplace-based Assessments and the College
WBA is part of assessment system for Specialist Trainees from 1st August 2007
Assessments need to be undertaken from early in training and recorded for ARCP
2007 – Launch of tools and paper-based assessment forms
College is researching web-based systems similar to the ones used in foundation
Workplace-based Assessments and the College
April 2008 – pilot web-based system By August 2008 – launch of web-based
system Use data generated by all assessments
including MSF and OSPE for evaluation of assessment strategy
Ensure compliance with MMC Gold Guide and planned ARCP
Vision for College Assessment system College will have integrated specialty
curriculum and assessment system endorsed by PMETB
College will have fully validated workplace-based assessment tools delivering feedback on progress to trainees throughout course
Valid and reliable tests will underpin MRCPath examination – Part 1 test of knowledge, Part 2 test of applied knowledge and skills
Factors affecting future assessment strategy
Selection into specialty may not be an annual process in future compromising ability to deliver year 1 assessment (except maybe in Histopathology)
Multiple recruitment may require more flexibility in arrangements for WBA
Selection into specialty may involve a national set of tests (Tooke and Academy) displacing Year 1 assessment
Factors affecting future assessment strategy
Workplace-based assessment is likely to continue but its use must be evidence-based
Workplace-based assessment is not cast in stone – work in progress and subject to improvement following evaluation
Place of individual components of assessment in MRCPath, Year 1 assessment and workplace-based assessment may change
Summary The College’s education and training, assessment
and examinations departments have worked hard to produce curricula and assessments acceptable to PMETB
This training day is the first for workplace-based assessment and is intended to assist training centres to develop assessment locally
Workplace-based assessment will be subject to a full programme of validation and optimisation to ensure fitness for purpose
The College is looking to source a web-based system to support workplace-based assessment
Glossary
ARCP Annual Review of Competence ProgressionCbD Case-based Discussion DOPS Direct observation of practical skillsECE Evaluation of clinical events MSF Multi-source feedback (ePATH-SPRAT)Mini-CEX Mini clinical evaluation exercise OSPE Objective structured practical examination PMETB Postgraduate medical education and training boardWBA Workplace-based assessment
top related