kwilby qs maple may 5 2015 · exam development • standards set to establish ‘cut scores’ for...

ACHIEVING PROGRAM LEARNING OUTCOMES THAT ALIGN WITH INTERNATIONAL ACCREDITATION STANDARDS Overcoming contextual and cultural barriers to assessment Dr. Kyle John Wilby, BSP, ACPR, PharmD Assistant Professor of Clinical Pharmacy and Practice Coordinator of Assessment and Accreditation College of Pharmacy, Qatar University, Doha, Qatar

1 May 5th 2015, Kyle John Wilby; QS MAPLE

Outline 1.  Background and Rationale 2.  Use of Cumulative Assessment to Demonstrate

Achievement of Program Learning Outcomes a.  Blueprinting

b.  Exam Development

c.  Required Training

d.  Implementation Challenges

e.  Stakeholder Buy In

3.  Recommendations and Lessons Learned

May 5th 2015, Kyle John Wilby; QS MAPLE 2

Background and Rationale • College of Pharmacy, Qatar University gained full

accreditation from the Canadian Council for Accreditation of Pharmacy Programs (2012-2018)

• Entry-to-practice exams implemented in Canada do not exist to the same extent in Qatar

• Accrediting body required a rigorous exit-from-degree assessment method to ensure PLOs achieved


Cumulative Assessment in Qatar • College of Pharmacy, Qatar University sought expert

consultation from Canada to design a cumulative assessment

• Emergence of a partnership with Supreme Council of Health •  To determine feasibility of examination methods for health

professionals in Qatar


Organizational Structure • Steering Committee

•  2 Expert Consultants (UT)

•  3 Chief Administrators (QU/SCH)

•  2 Chief Examiners (QU)

•  Joint Committee •  Steering Committee

•  Administrators from QU, HMC, PHC, SCH


Key People: •  Admin support (QU/SCH) •  Prof. Lab Tech (QU)

Assessment Structure

Knowledge Application Performance


Multiple Choice Questions (MCQ)

Written Care Plan Objective Structured Clinical Examination

(OSCE)

N = 23 Graduating Pharmacy Students

Blueprinting


• PLOs adopted from Association of Faculties of Pharmacy of Canada (AFPC) Educational Outcome for First Professional Degree Programs in Pharmacy

•  Two local faculty members embedded within practice reviewed for contextual adaptation •  Two competency statements removed during this process due to

inapplicability to Qatar’s practice model

• Pharmacy curriculum blueprinted to total amount time spent on each system of therapeutics (i.e. respiratory)

Exam Blueprint MCQ Care Plan OSCE Competency Total* 53% 70% 45% Care Provider 53.25% 2% 10% 40% Communicator 23% 5% 0% 5% Collaborator 3.75% 13% 0% 0% Manager 3.25% 5% 0% 5% Advocate 3.75% 15% 20% 0% Scholar 8.75% 7% 0% 5% Professional 4.25%


*Total percentages benchmarked against the Pharmacy Examining Board of Canada’s blueprint for the Qualifying Examination

OSCE Blueprint - Complexity

Simple Patient-Simple Problem (50%)

Simple Patient-Complex Problem (25%)

Complex Patient-Simple Problem (25%)

Complex Patient-Complex Problem (0%)


Exam Development 1.  Knowledge Assessment

•  Case-based MCQs developed by collaborative groups of faculty members

2.  Application Assessment •  Two paper patient cases developed by Chief Examiners and

reviewed for complexity by peers and expert consultants

3.  Performance Assessment •  Case development and validation with faculty and practitioners


Exam Development • Standards set to establish ‘cut scores’ for each

component to act as pre-defined level of competency

• OSCE overall cut score determined by adding each station’s cut score

• Care Plan cut scores determined post-hoc through a faculty focus group to create rubric and determine passing standards


Standard Setting • Anghoff Method

•  “Out of 100 minimally competent graduates from the QU-CPH BSc (Pharm) program, how many would you expect to achieve this point?”


90

50

60

Standard Setting • Anghoff Method

•  “Out of 100 minimally competent graduates from the QU-CPH BSc (Pharm) program, how many would you expect to achieve this point?”


60

OK!

Care Plan Rubric


Training • Perhaps the most important component to ensure

examination validity •  Training of exam developers (all three components)

•  Assessor training for OSCE stations

•  Standardized patient training for OSCE stations (amateur actors)

•  Ensuring standardized grading for application component


Assessor and Standardized Patient Training •  Mandatory 2-3 hour sessions •  Overview of exam logistics and procedures •  Review of two cases for assessors to evaluate and

discuss differences in grading •  Portrayal of emotions for SP training

OSCE - Showtime!


•  24 Students + 24 Practicing Pharmacists + 1 Control •  3 tracks, 2 cycles

•  59 Assessors

•  45 Standardized Patients

•  30 Exam Center Staff

•  5 Steering Committee Members

• Knowledge and Application Exams

Implementation Challenges 1.  Standard setting and cut scores 2.  Recruitment and training of standardized patients

3.  Reliability of assessor judgments

4.  Validity of assessment tools in context

5.  Capacity building and knowledge sharing


Inter-rater Reliability


Topic Complexity* Analytical ICC Global ICC Osteoporosis S-S 0.62 0.85 Endocrinology S-S 0.92 0.81 Infectious Dis. S-C 0.81 0.24 Cardiology C-S 0.68 0.36 Pain C-S 0.87 0.74 Asthma (device) S-S 0.56 0.30 Depression S-C 0.96 0.55

*Problem-Patient; S = Simple, C = Complex

Pharmacist Reliability


Topic Complexity* Analytical ICC Global ICC Osteoporosis S-S 0.92 0.77 Endocrinology S-S 0.88 0.81 Infectious Dis. S-C 0.87 0.42 Cardiology C-S 0.92 0.62 Pain C-S 0.88 0.22 Asthma (device) S-S 0.92 0.64 Depression S-C 0.92 0.78

*Problem-Patient; S = Simple, C = Complex


Is it valid to assess communication skills according to Western standards?

Stakeholder Buy In • Essential for success of any major assessment

•  Students

•  College administration

•  University administration

•  Government

•  Community / Practitioners

•  Accrediting organization


Overall Summary and Recommendations • A multi-component exit-from-degree exam is feasible in

our setting •  Refinement required to improve validity and reliability

•  Future iterations will build capacity and expertise within Qatar

•  Careful attention should be paid to contextual factors differing from North American settings (standard setting, prompts, etc.)

•  Greatest limitation was standardized patient performance due to use of amateur actors with little (if any) experience

•  Future iterations to blueprint to individual competency statements for quality assurance and program improvement


Thank You


kwilby qs maple may 5 2015 · exam development • standards set to establish ‘cut scores’ for...

Documents