2007 annual meeting medical school assessment: role of high stakes external examinations november 4,...

2007 ANNUAL MEETING

Medical School Assessment: Role of High Stakes External Examinations

November 4, 20073:45 – 5:00 PM

Health in the Balance

2007 ANNUAL MEETING

ParticipantsRuth B. Hoppe, MD Michigan State Univ.,

Chair, USMLE Composite Committee

Peter Scoles, MD Senior Vice President for Assessment

Programs, NBME Alfred F. Tallia, MD, MPH RWJ Medical School

Chair, CEUP Committee


2007 ANNUAL MEETINGParticipants, cont.

Kim E. LeBlanc, MD, PhD Louisiana State University, Head, Department of Family Medicine

Molly Osborne, MD, PhD Oregon Health & Science University, Associate Dean for Student Affairs

Ruth-Marie E. Fincher, MD Medical College of Georgia, Vice Dean for Academic Affairs


2007 ANNUAL MEETINGOur Plan for Today

•Description of the CRU process – Drs. Scoles & Tallia

•Perspectives from the Panelists

Dr. LeBlanc Licensure perspective

Dr. Osborne Student Affairs perspective

Dr. Fincher Academic Affairs perspective

•Questions/Comments from the Audience – Pass Cards to Aisles


2007 ANNUAL MEETING

Comprehensive Review of USMLE (CRU)

Background For Today’s Discussion

Peter Scoles, MD

Senior Vice President for

Assessment Programs, NBME


2007 ANNUAL MEETINGUSMLE

Primary mission is certification of individual knowledge and skill for licensure

Important secondary uses:

• Promotion and graduation decisions

• Curriculum evaluation

• One factor in residency selection process

• International graduates must pass Step 1, Step2 CK, & Step 2 CS for ECFMG certification


2007 ANNUAL MEETING

USMLE Architecture designed in 1980’s

Academic and practice environments have changed

We’ve heard that:

• Separation of basic science and clinical science is artificial

• Current structure impedes curriculum innovation

• Knowledge is only one of the competencies necessary for practice

• There is unwarranted dependency on scores for the NRMP


2007 ANNUAL MEETING

Although it is possible that the current structure is the most effective and efficient method to meet the needs of all stakeholders, it seems prudent to conduct a comprehensive review of the program


2007 ANNUAL MEETINGGoals of USMLE Review

A data-driven, iterative process that provides the Composite Committee and parent organizations with a complete picture of the strengths and weakness of the current program, and recommendations, if any, for change


2007 ANNUAL MEETING

We are early in the course of a process that may take 4 or more years to develop and implement:

•2004 Composite governing committee charges NBME, FSMB, ECFMG to begin comprehensive review of USMLE

•Early 2006 Committee to Evaluate the USMLE Program (CEUP) formed


2007 ANNUAL MEETINGCEUP Committee

Membership:

• students, residents, fellows, deans & associate deans, basic science & clinical faculty, international medical graduates, state medical board members, practicing physicians, & the public.

• some (but not all) have experience in USMLE program as item writers, reviewers, or test committee members.


2007 ANNUAL MEETINGCEUP Process

Develop principles to guide deliberations

Examine data in depth

Determine what changes, if any, needed

Consider different models

Engage stakeholders in iterative discussion

Formulate final recommendations


2007 ANNUAL MEETING

Comprehensive Evaluation of USMLE Program

ALFRED F. TALLIA, MD, MPHUniversity of Medicine and Dentistry of New Jersey

Robert Wood Johnson Medical School

CHAIR, CEUP


2007 ANNUAL MEETINGQuestions

What is the role of a high stakes examinations in the competency assessment of physician candidates for primary licensure?

What competencies should be assessed? By what means?

How is the current process working? Are there competencies that should be assessed that currently aren’t?


2007 ANNUAL MEETING

Guiding Principles

1) First priority of USMLE: assure licensing authorities that candidate possesses knowledge & skills for safe & effective patient care in both supervised & unsupervised settings

2) Secondary uses of USMLE results should be supported as long as primary purpose of USMLE not compromised

3) For all uses, USMLE assessment instruments should be valid & reliable measures of competencies required for medical practice

4) USMLE must continue to reflect evolving national consensus of competency


2007 ANNUAL MEETINGData Acquisition Phase

to inform CEUP process

surveys & focus groups

information about impact & relevance

wide range of individuals & organizations

• representatives from medical licensing authorities & US undergraduate & GME communities

• institutional & national leaders from international medical education community

• recent USMLE examinees, both US-trained & internationally-trained

• leaders of local & national student groups


2007 ANNUAL MEETINGTheme #1

At entry into graduate training, doctors must have minimum competency in basic clinical knowledge & those skills necessary to safely care for patients.

At time of licensure, higher level of these competencies, together with others acquired during GME, are necessary.

If these competencies can be measured in valid, reliable, & practical manner, they should be incorporated into the USMLE.


2007 ANNUAL MEETINGTheme #1 cont.

Licensure examination program should be more explicitly designed to support decisions at two points.

• First, a decision about readiness to begin direct patient care under supervision, at interface between undergraduate & graduate medical education.

• Second decision relates to readiness for physician to provide unsupervised patient care & to obtain license for unrestricted practice.


2007 ANNUAL MEETINGEvolving USMLE Competencies

ACGME/ABMS, National Alliance for Physician Competence (ECFMG, AAMC, NBME, FSMB, ACGME, ABMS), Good Medical Practice-USA • Patient Care• Medical Knowledge & Skills• Practice-based Learning and Improvement• Interpersonal & Communications Skills• Professional Behavior• Systems-based Practice



Separate design & administration of examinations of basic & clinical sciences creates an artificial separation.

Weight of opinion favors substantive integration of basic & clinical science concepts throughout all examination components.

Widely expressed sentiment, including both basic & clinical sciences faculty members.



Current Step 1 used by many medical schools to support promotion or graduation decisions.

If USMLE redesign eliminates Step 1, NBME should provide similar valid, reliable, & secure assessment tools to schools that still wish to use them.



Conflicting opinions on value of numeric versus pass/fail reporting for both primary & secondary uses.

Issue not resolved until final recommendations made & implications fully examined.



USMLE as a high stakes examination must be viewed in the context of lifelong learning and competency assessment.

There may be new essential medical competencies that will arise; USMLE design must be sufficiently flexible to incorporate these to the extent possible.


2007 ANNUAL MEETINGProposed ModelGateway 1• Supervised medical practice• Integrative and competency based• Includes tests of knowledge and clinical skills• Content may expand to include other knowledge relevant to

practice of medicine, e.g. social sciences, health policy

Gateway 2• Unsupervised, independent medical practice• Integrative and competency based, higher order competencies• May include new testing modalities under development that

assess evolving skill sets, e.g. cultural competency, communication skills, systems based practice


2007 ANNUAL MEETINGMisconceptions

Misconception: recommendations involve combination of current Step 1 & 2, & subsequent administration of current Step 3. NOT the case.

Instead, we anticipate building new test items that measure not only mastery of current basic science information, but also ability to deal with emerging concepts with future relevance for the practice of medicine.

Both "gateways" would contain these materials.


2007 ANNUAL MEETING

Licensure Perspectives

Kim Edward LeBlanc, MD, PhD

• Marie Lahasky Professor and Head, Department of Family Medicine, Louisiana State University School of Medicine – New Orleans

• Member, Board of Directors, Federation of State Medical Boards

• Member and Past President, Louisiana State Board of Medical Examiners

• Member, USMLE Step 3 Committee


2007 ANNUAL MEETINGLicensure Perspective

State Medical Boards (SMBs) concerned about opening their Medical Practice Acts

• Most MPAs do not specifically reference USMLE in their statute(s)

• Most use “examination acceptable to board”

• 49 of 70 SMBs issue resident licenses Minority (16) mandate Steps 1 & 2

If changes to USMLE require statutory relief at state level, may take up 18-24 months to accomplish this



Numeric scores vs. pass/fail• Some SMBs determine passing score• Different constituencies have very different and

strongly held beliefs ~ 84% SMBs prefer numeric scores US and IMG examinees: scattered preferences US Deans have scattered preferences ~ 93% of residency directors prefer numeric

scores

• USMLE first and foremost a licensing exam



Competencies

• General undifferentiated medical practice

• Not specialty based or practice focused

• Strong foundation of medical knowledge essential

• Also very important professionalism communication skills ability to research, gather, and apply information



Clinical Skills

• Critically important for unsupervised practice

• SMBs exist to protect the citizens of that state Need to be as certain as possible that the

licensee is competent and capable

Timing of the examinations

• Lengthening the licensure process? Current process working well with 3 steps


2007 ANNUAL MEETINGStudent Affairs Perspective

MOLLY OSBORNE, MD, PhDOregon Health & Science University

VICE CHAIR

AAMC GROUP ON STUDENT AFFAIRS


2007 ANNUAL MEETINGStudent Affairs Perspectives

Positive:Longitudinal integration of basic science and

clinical curriculum

Opportunities to integrate ACGME core competencies into a 4 year curriculum

Opportunities for courses including first through fourth year students

Opportunities to develop ‘pilot’ curricula An interactive process from bench to

bedside and back



Unintended consequence:

One attempt at a high stakes examination

• High anxiety

• If poor USMLE 1 score, no chance for improved score on USMLE 2

• Single value for Program Directors use Often used for interview selection

(Waggoner et al Acad Med 1999)

• Single value for career advising



Steps to overcome challenges:

• NBME should provide similar valid, reliable, & secure assessment tools

• NBME must provide rapid ‘turn around’ for reporting of scores, or Pass/Fail

• NBME [FSMB] should perform outcomes research to track associations of USMLE scores and physician performance



Steps to overcome challenges cont.:

•Need for education and evaluation to interact so that evaluation doesn’t drive education

•Clinical and Basic Science Faculty development

•Improved transparency between Program Directors and Advisors


2007 ANNUAL MEETINGAcademic Affairs Perspective:UGME and GME

RUTH-MARIE E. FINCHER, MD

Vice Dean for Academic Affairs

Medical College of Georgia School of Medicine

Former Chair, AAMC Group on Educational Affairs

NBME Board Executive Committee member


2007 ANNUAL MEETING

Dr. Fincher:Academic Affairs (UME, GME)

Positives

• Enable innovative curriculum development Remove artificial distinction between basic

science and clinical medicine

• Foster students’ integrative learning and basic science application across 4 years


2007 ANNUAL MEETING

Academic Affairs - UGME - Concerns

Loss of external benchmark

• Pass required for promotion to year 3 in many schools

VERY high stakes – increased student anxiety

MD/PhD students

• Far from beginning medical school


2007 ANNUAL MEETINGAcademic Affairs - UGME - Concerns

Basic science

• Undesirable diminution of basic science emphasis

“Force” curriculum integration

Clinical skills assessment – When?


2007 ANNUAL MEETINGAcademic Affairs - GME - Concerns

Negative impact on:

• Interviews, if scores unavailable by fall of Yr 4

• Rank order, if scores unavailable by Feb. Yr 4

• Graduation and starting residency if fail

Scoring

• Many PDs want percentiles for comparison

• Many comments against pass-fail

Possible delay in timing of Gateway B

• Impact on licensure, early specialization


2007 ANNUAL MEETINGPotential Responses to Concerns

NBME could provide non-licensure exam similar to Step 1 that schools could use for promotions decisions

Create “mock” gateway sample items for review

Basic science emphasis likely to increase as thread through 4 years

• Curriculum, teaching, and assessment implications

Gateway B at end of PGY-1, like Step 3


2007 ANNUAL MEETING Discussion

Your Thoughts, Comments, Questions


2007 ANNUAL MEETINGNext Steps

Continued iterative discussion and input to CEUP

CEUP to formulate final recommendations

Ratification by NBME, FSMB, and USMLE Composite Committee

Extended period of design and pilot work prior to implementation


2007 ANNUAL MEETING Opportunity for Further Input

www.usmle.org/comprev

2007 annual meeting medical school assessment: role of high stakes external examinations november 4,...

Documents