jocelyn lockyer phd senior associate dean, education
DESCRIPTION
The Physician Achievement Review Program: A collaboration between the College of Physicians and Surgeons of Alberta and University of Calgary. Jocelyn Lockyer PhD Senior Associate Dean, Education Professor, Department of Community Health Sciences University of Calgary. Background/Disclosure. - PowerPoint PPT PresentationTRANSCRIPT
Jocelyn Lockyer PhDSenior Associate Dean, Education
Professor, Department of Community Health Sciences
University of Calgary
1
No financial conflicts of interest to report College of Physicians and Surgeons of Alberta—
Physician Achievement Review Program ◦ Since 1996◦ Instrument development, testing, and
assessment
2
CollaboratorsCPSA
◦ John Swiniarski, Bryan Ward (deceased), Trevor Theman
◦ Physician Performance CommitteeUniversity of Calgary
◦ Research Team..Claudio Violato, Herta Fidler◦ Physician Advisor..Ray Lewkonia
Pivotal Research Inc◦ Steve Dennis
3
◦ Describe the Alberta Physician Achievement Review (PAR) Program
◦ Discuss the evidence supporting PAR as a tool for screening and quality improvement
◦ Describe dissemination and uptake by others within and outside Canada
◦ Provide an opportunity to answer questions about PAR
4
Multisource feedback program (aka 360 degree evaluation)
Questionnaire based assessment Feedback about observable behaviors is
obtained every 5 years from◦ 8 Physician colleagues (peers, referring MDs,
referral MDs)◦ 8 Co-workers (e.g., nurses, pharmacists,
dieticians)◦ 25 Patients ◦ Self
5
6
MSFSelf-assessment
Co-workers
Medical colleagues
Patients
First level screen which might◦ Provides physicians with data about their practices to
guide improvement◦ Identify need for further assessments
Source of unique and useful data that can inform many of the CanMEDs roles, particularly◦ Communicator◦ Collaborator◦ Professional◦ Medical expert
Relatively inexpensive Adaptable Repeat administration (5 years)
7
8
General/family physicians (revised 2010)
Surgical specialists Medical specialists
◦Pediatrics◦Psychiatry◦All other medical specialists
Anesthesiologists “Episodic care” providers Diagnostic Imaging Laboratory Medicine
PAR Survey
Self-Analysis
Response
PARCommittee
Review
IndividualizedAssessment
Advice toPhysician
Feedback
Office Peer Review
CompetenceAssessment
Diagram 1
The College’s PAR Program
9
Good assessment should offer1. Validity or coherence2. Reproducibility or consistency (reliability)3. Equivalence4. Feasibility5. Educational effect6. Catalytic effect7. Acceptability
Norcini et al, Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference, Medical Teacher 2011
10
11
Criteria Evidence
Validity Developed by physicians with input from physicians, health care professionals and patients.Factor analyses showing items correlate in intended ways to form scales.Positive but low correlation between sources.
Reliability 22-25 patients, 8 colleagues and 8 co-workers for reliability suitable for formative (lower stakes) assessment based on G- and D-studies.
Equivalence Comparisons in Nova Scotia and Alberta.Comparisons by different schools of graduation.
Acceptability High response rates from all sources.>70% of physicians report obtaining valuable feedback.
12
Criteria Evidence
Educational effect
49% report making a change in practice as a direct result of PAR. Most changes on patient care, communication with patients, colleagues, and co-workers and stress managementImprovement in scores when tested 5 years apart.
Catalytic effect Not studied.
Total 9816
Family medicine 4355
Surgeons 1161
Anesthesiologists 472
Medicine (and subspecialties) 1506
Pediatricians 442
Psychiatrists 617
Episodic care 842
Diagnostic imaging 271
Laboratory medicine 150
13
Dissemination◦ >25 publications in peer review journals
Adopted◦ Nova Scotia (NS-PAR)◦ Manitoba (M-PAR)
Being Tested◦ British Columbia◦ Council of Teaching Hospitals of Ontario◦ Royal College of Physicians and Surgeons of Canada
Practice eligible route Tested
◦ Medical Council of Ireland◦ Netherlands (Internal Medicine)◦ Singapore (Emergency Medicine)
14
Questionnaire based assessment in which feedback about observable behaviors is provided by other physicians, co-workers (e.g., nurses, social workers), and patients
Program that meets criteria for a ‘good assessment’ Initiative that can be adopted and used by others but
attention needs to be paid to◦ Purpose
Quality improvement/formative assessment◦ How it functions in different settings with special reference to
how physicians use the data to inform practice change
15
16