aamc activities amsndc cas representation

AAMC ActivitiesAMSNDC CAS Representation

Presentation to the 2008 AMSNDC Meeting, St. Croix, USVI, March 7, 2008

By: Mike Friedlander


2004-08 – Member, Council of Academic Societies (CAS) Administrative Board

2006-07 – Chair, CAS2007- AAMC CAS Spring Meeting on Role of Research in Medical Education2007-08 – Immediate Past Chair, CAS2006-07 – Member, AAMC Executive Committee2006-08 – Member, AAMC Executive Council2006-08 – Member, AAMC Governance Review Team2007-08 – Association of Professors of Medicine (APM) Task Force on

Revitalizing the Nation’s Physician-Scientist Workforce2006-08 – AAMC CAS Liaison to the Group on Graduate Research, Education

and Training Steering Committee (GREAT)2006-08 – Member, HHMI-AAMC Panel on the Scientific Basis of Medical

Education2006-08 – AAMC Task Force on Industry Support for Medical Education2007 – Organizer, AAMC Symposium on the Scientific Basis of Influence and

Reciprocity2007-08 – Advocate, USMLE-NBME Program Review Process



2006-07 – Chair, CAS2007-08 – Immediate Past Chair, CAS2006-07 – Member, AAMC Executive Committee2006-08 – Member, AAMC Executive Council2006-08 – Member, AAMC Governance Review Team2007 – 08 – Association of Professors of Medicine (APM) Task Force on

revitalizing the Nation’s Physician-Scientist Workforce2006-08 – AAMC CAS Liaison to the Group on Graduate Research, Education

and Training Steering Committee2006-08 – Member, HHMI-AAMC Workgroup on the Scientific Foundation for

Future Physicians2006-08 – AAMC Task Force on Industry Support for Medical Education2007 – Organizer, AAMC Symposium on the Scientific Basis of Influence and






and Training Steering Committee2006-08 – Member, HHMI-AAMC Workgroup on the Scientific Foundation of

Future Physicians2006-08 – AAMC Task Force on Industry Support for Medical Education2007 – Organizer, AAMC Symposium on the Scientific Basis of Influence and


HHMI-AAMC Scientific Foundation for Future Physicians

Co-Chairs Bob Alpern, MD, Dean, Yale University School of MedicineSharon Long, PhD, Prof of Biol. Sci. and Dean of Humanities and Sciences, Stanford

MembersManuel Ares, PhD, Prof & Chair, MCDB, UC Santa CruzDennis Ausiello, MD, Prof of Clinical Med, HarvardJudith Bond, PhD, Prof & Chair, Biochem, Penn StateArthur Dalley, PhD, Prof, Dev & Cell Biology, Vanderbilt Med SchoolJulio de Paula, PhD, Prof Chem, dean of Arts & Sciences, Lewis & Clark Univ.Andrew Fishleder, PhD, Chair, Div of education, Cleveland Clinic Fndn.Mike Friedlander, PhD, Prof & Chair of Neuroscience, Baylor College of Med.Gary Gibbons, MD, Prof Medicine, Dir CV Res. Inst., Morehouse Med SchoolRobert Hilborn, PhD, Prof Physics, Dir Ctr for Sci, Math and Comp. Ed, U Nebraska John Holmes, PhD, Asst Prof Medical Informatics, U. PennPaul Insel, MD/PhD, Prof Pharm/Med, Dir MSTP UCSDLynn Kirk, MD, Assoc Dean for Grad Med Ed, UT SouthwesternBruce Korf, MD/PhD, Prof-Chair of Human Genetics, UABVinay Kumar, MD, Prof & Chair, Pathology, U. ChicagoDeidre LaBat, PhD, Prof of Biology, Xavier Univ.Paul Marantz, MD, Prof of Clinical Epidemiol., Albert Einstein, Yeshiva U.Claudia Neuhauser, PhD, HHMI Prof of EEE, U. MinnesotaGreg Petsko, PhD, Prof of Biochem & Chem, Brandeis U.Robert Siegal, MD/PhD, Assoc Prof of Teaching Dept. of Microbiol & Immunol, Stanford Med SchoolDee Siverthorn, PhD, Senior Lecturer, School of Biolog. Sci., U. of TexasAlistair Wood, MD, Managing Director, Symphony Capital LLC





and Training Steering Committee2006-08 – Member, HHMI-AAMC Panel on the Scientific Basis of Medical



Task Force on Industry Funding of Medical Education

Co-ChairsRoy Vagelos, MDBill Danforth, MD

MembersAcademic Medicine (Deans, Chancellors, Chairs, Basic Scientists, Clinicians, Residents, Students)Industry (Pharmaceutical and Medical Device – Amgen, Pfizer, Medtronics)PolicyEthicsAAMC – David Korn, MD, Susan Ehringhaus, JD

18 months meetings, work groups - Professionalism and Med Ed, Benefits & Pitfalls, Unmet Needs & Opportunities – Final Meeting 2/26/08;

Charge to the Task Force• Review range of policies and procedures currently in place in medical

schools and teaching hospitals for managing industry support of educational activities and industry gifting practices directed at students, residents, faculty and staff;

• Evaluate benefits to be gained and pitfalls to be avoided in the relationships between industry and academic medicine;

• Develop general principles to guide academic medical institutions in optimizing the benefits and minimizing the pitfalls of industry support for medical education;

• Identify educational strategies currently used by academic medical centers to raise awareness about the benefits and pitfalls of industry support for medical education, and whether any of the strategies have been shown to be effective;

• Suggest the scope of ongoing work that the AAMC might undertake to assist its members to operationalize the general principles articulated by the task force.

• Speakers– Dan Ariely, PhD, Prof of Behavioral Economics, MIT

– Max Bazerman, PhD, DSc, Professor of Business Administration, Harvard Business School, Prof of Psychology, Kennedy School of Government

– Mike Friedlander, PhD, Professor of Neuroscience, Director of Neuroscience Initiatives, Baylor College of Medicine

– David Korn, MD, Sr. VP, Division of Biomedical and Health Sciences Research, AAMC

– George Loewenstein, PhD, Professor of Economics, Psychology, Carnegie Mellon University

– Read Montague, PhD, Professor of Neuroscience, Director Human Neuroimaging Lab, Baylor College of Medicine

Panel

– David Blumenthal, MPP, PhD, Director, Institute of Health Policy, Mass General, Prof of Medicine, Harvard

– Harry Greenberg, MD, Assoc. Dean for Research Stanford U. School of Med.– Tom Murray, PhD, President, The Hastings Center– Bruce Psaty, MD, PhD, Director of CV health Research Ctr, U. Washington– David Rothman, PhD, President, Institute on Medicine as a Profession, Professor of Social

Medicine, Columbia U.

Presentation to the AAMC COD Ad BoardFebruary 13, 2008

“Issues of Interest/Concern to the Academic Medicine Community Regarding

the Proposed Changes to the USMLE”

Mike Friedlander, PhDBaylor College of Medicine

Past Chair, AAMC CAS

David Engman, MD, PhDNorthwestern University Feinberg School of Medicine

Chair, AAMC MD-PhD Section

Role of science in medicine in the twenty first century

• Likely more (vs. less) important than in the twentieth century

• Genetics, genomics, proteomics, biological basis of complex behaviors and behavioral disorders

• Informatics, critical reasoning skills, life-long learners, ability to discern “pitch” vs. information; going beyond algorithmic approach; dealing with the unexpected

• Understanding how data are obtained and weighing competing claims; evidence based medicine

• Stated goals of LCME and MSOP IV:

– The curriculum must include current concepts in basic and clinical science, underlying scientific concepts of medicine, contemporary content, critical analysis of data, foster intellectual challenge

– . . . must acquire knowledge and understanding of the principles of the biomedical sciences and how new knowledge is developed

• Differentiating the US allopathic physician; leadership in team medicine

• The public trust; competency to discuss/explain the scientific rationale of diagnosis/treatment to the patient and in context of readily available information (the educated patient)

• Composite committee requests NBME develop process to review the USMLE

• Committee to evaluate the USMLE (CEUP) formed - begins evaluation process

• Proposal to modify the USMLE posted on NBME site

• Proposal rolled out to academic medicine and education communities

• Feedback on proposal solicited from various stakeholders including discussion at AAMC annual meeting

• Various clinical and biomedical science communities begin considering and responding to proposal

• NBME hosts meeting to consider suggestions/concerns

• Modified proposal posted on NBME site

• NBME extends meetings with stakeholders (FASEB, GREAT, CAS, etc)

• CEUP recommendations made to composite committee

Background

• Program has been unchanged without major evaluation for a long time

• Artificial divide between basic and clinical science on USMLE exams

• Need to extend learning of scientific principles of medicine throughout the medical education continuum

• Poor performance of recent graduates on analytical questions (data interpretation/analysis)

• “Binge and purge” behavior on learning basic science principles

• USMLE step 1 exam at end of second year incentivizes MS IIs to opt out of formal curriculum to study for step 1 exam

• Exam structure prevents curriculum innovation and hinders schools for customizing their curricula to their unique missions

Motivation for evaluation

• Two decision points – 1) readiness for patient care under supervision; 2) readiness for unsupervised patient care;

• For post graduate training, minimum competency in basic clinical knowledge and skills to provide safe care; incorporate the measurement of these competencies in the USMLE;

• Separate exam on basic sciences seems to create an artificial separation of basic and clinical sciences; weight of opinion favors integration of basic and clinical science concepts throughout all exam components;

• Current step 1 used for promotion and graduation decisions – if step 1 eliminated, NBME should provide similar assessment tools to schools who want to use them;

• Numeric vs. pass/fail reporting for primary and secondary uses of USMLE issue not resolved.

Original proposal posted August 15, 2007

• NBME officers (Drs. Scoles, Melnick and Dillon) hold numerous meetings with various stakeholder groups

• Meeting with basic science and clinical representatives on January 3, 2008 in Philadelphia

– NBME officers realize that basic science and clinical societies not adequately consulted

– Appreciate that the step 1 exam has “intrinsic value” as a pedagogical tool

– Ask meeting participants to encourage AAMC to convene meeting of “stakeholders” to discuss the broader implications of exam

• NBME officers modify recommendation based on feedback from various groups

Activities between August and February 2008

• The Association of Professors of Medicine

• The American Society for Clinical Investigation

• The American Society for Pharmacology and Experimental Therapeutics

• The American Physiological Society

• The Association of University Professors of Neurology

• The American Society for Biochemistry and Molecular Biology

• The National Association of MD-PhD Programs

• The Association of Medical School Neuroscience Department Chairs

• The American Society for Nephrology

• The American Association for the Study of Liver Diseases

• The American Gastroenterological Association Institute

Associations that have expressed concern that basic science content in the exam will be reduced

• CEUP has completed its review of information gathered during early phases; mid-March, 2008 targeted to complete final recommendations

Final report will likely reflect:

• Assessments to inform state licensing authorities at two “gateway” points (entry to supervised practice and entry into unsupervised practice); possible that there could be more than one assessment for each gateway

• Redesign of the USMLE to better reflect competencies important to medical practice to the degree they can be measured

• Reconsider the independent assessment of the basic sciences in favor of an integrative approach

Revised proposal posted February 1, 2008

• Most students see value in studying for USMLE step 1 and say that test has intrinsic value:

– studying for step 1 is “scholarly” activity that involves (i) review of basic mechanisms of human health and disease (ii) filling in the gaps, (iii) integrating knowledge across levels (molecules to cells to organs to organism)

– step 1 facilitates their understanding of the basic scientific principles that underlie the molecular, cellular and organismic basis of health and disease

– the group that feels least strongly that the step 1 exam has intrinsic value is the 2000 second-year students currently preparing to take the exam

• Survey will be completed later in February

Ongoing survey of ~8000 US medical students

• Consideration of three parts to Gateway A exam, including possibility of one that evaluates basic scientific knowledge/competency, one that evaluates ability to understand and evaluate data and one that evaluates clinical skills/knowledge

• Possibility for flexibility in timing of exam parts for individual schools

• Consideration of possibility of independent non-compensating requirements in test components

• Consideration of possibility of value of exam early in educational process to evaluate the student’s ability to integrate across the sciences and to provide an opportunity to study/fill in knowledge gaps

• Consideration of role of secondary uses of exams by schools

Recent progress

• Report has considerable flexibility for how importance of science is weighted and how testing implemented

• Many Societies/Associations (especially clinical) are only now getting opportunity to evaluate implications and weigh in

• Integrating basic science across the medical curriculum

• Defining basic science relevant to the practice of medicine

• Objective evaluation of outcomes and alternative models

• Message to students, curriculum committees, public

• Leadership role of AAMC in process

Challenges and opportunities ahead

What to do

• Engage with our deans over the next 3 weeks on the subject

• Volunteer yourself or nominate faculty to NBME to serve on policy panels and for USMLE development

• Take back an active role in scientific medical education; recognize and reward your faculty; don’t fall prey to outstanding scientists don’t/can’t/won’t/shouldn’t teach

• Work together with our clinical colleagues to help redefine the scientific basis of allopathic medicine in the US and differentiate the practice of medicine from “algorithmic only” “competencies.”

• Make the case for the importance of teaching the scientific principles of medicine as good for medicine and the public health in the US – not protecting turf of the basic science community.

• Don’t whine if we don’t participate and contribute!

Changes to USMLEPeter V. Scoles MD

[email protected]

USMLE

• Primary mission is certification of individual knowledge and skill for licensure

• Important secondary uses– Promotion and graduation decisions– Curriculum evaluation– One factor in residency selection process – International graduates must pass Step 1, Step2 CK, &

Step 2 CS for ECFMG certification

USMLE designed > 20 years ago

Basic science largely in Step 1Most items have associated clinical vignettes,

but these often are “window dressing”“Integrated” items are difficult to write

Clinical knowledge and skills in Step 2Step 3 lacks relevance

Rotating internships uncommonFew physicians enter practice after 1-2 years

Little emphasis on

Evidence based practiceGathering and interpretation of

informationBiostatistics/epidemiology/

Public HealthCost effective practiceRehabilitation

Although it is possible that it is still the most effective and efficient method to meet the needs of all stakeholders, it

seems prudent to conduct a comprehensive review of the program

USMLE Parents

Composite Committee

Planning Committee

Committee to Evaluate USMLE (CEUP)

NBME, FSMB, ECFMG staff CEUP subcommittees

CEUP Report

March 2008

Stakeholder dialogue

Straw models, sample outlines and items

Prototype committee meetings

Internal staff feasibility reviews

Joint executive Boards

November 2008

Full Board meetings

Spring 2009

Final construct proposals

Review Timeline

Composite Committee review

June 2008

Guiding Principles

• Assure that USMLE meets the needs of state medical boards now and in immediate future

• Support legitimate secondary uses of the exam

• USMLE components should be valid & reliable measures of competencies required for medical practice

• USMLE must continue to reflect evolving national consensus of competency

Information Gathering Process

• Stakeholder surveys 2005-2006

• Regional Town hall meetings 2006-2007

• AAMC, GSE, GEA, OSR meetings 2006-2007

• AMA, SAMA 2006, 2007

• FSMB and SMB meetings 2006-2007

• Program director associations 2005-2007

• Special interest groups 2007-2008

State Medical Board/Public Perspective

Licensure decisions are necessary at two points:

1. Entry into supervised post graduate training

2. Primary licensure

At each point:

Measure all competencies related to patient centered care that can be tested in a valid and reliable manner

Assure at least minimum competency in these areas

For exam components, provide scores that inform state medical boards, especially when performance is marginal

Clinical Educators/Curriculum Officers/Student Affairs Deans

• Separation of basic science and clinical science in Step 1 and 2CK is artificial

• Step 1 interferes with curriculum design and delivery– Meaningful curriculum reform impossible

• Step 1 score disproportionately affects career decisions

Students

• Prefer to “get basic science over with”

• Not generally in favor of integrated exams

• Students in Years 3 and 4 prefer scores over pass/fail grading

• Students in Years 1 and 2 have mixed feelings about scores

• Women more likely than men to prefer integrated exams and pass/fail scoring

Basic Science Community

• Basic science is the foundation of medicine• Some basic science has no clinical “wrapping”• Step 1 reinforces the value of basic science• Studying for Step 1 is a scholarly activity • Step 1 provides a framework for learning• Step 1 is necessary for promotion decisions• Meaningful, nationally normed grades are

good

Educators and NBME agree that most candidates exhibit “Binge and Purge” behavior with regard basic science

foundations of medicine

-20

-15

-10

-5

0

5

10

1975 - 771981 - 841988 - 891991 - 922004 - 05

Total

Biochem

Micro

Anat PhysPharm

BehavSci Path

Basic Science Retention:

Post Graduate Community

• Scores are essential – In current system, must have Step 1 scores

no later than September 1st for interview screening

– Step 2CK and CS necessary for final ranking

• Structure and content of USMLE program is of secondary importance

Scores and Residency selection

• Most residency programs use Step 1 Scores as one factor in selection process

• This is a legitimate secondary use• Increasing pressure to provide scores for

use as a screening instrument for interview selection

• This has significant secondary consequences

Step 1 scores….

• provide a rapid way to screen applications

• are nationally normed

• Correlate with MCQ components of in- training and specialty certification exams– So do SATs and MCATs

On the other hand…

• Step 1 was not designed as an aptitude test– Criterion based rather than norm based scoring– Focal point is at “cut score”

• Evidence suggests that threshold for ability to pass specialty board exams is well below mean USMLE Step 1 performance

• Retention of Step 1 materials falls off rapidly • Steps 2 and 3, clerkship grades, and clerkship

director comments are better predictors of performance during residency

CEUP Themes

• An examination process which provides state medical boards with valid and reliable information for decisions regarding provisional and unrestricted licensure

• Informs decisions at two points or “Gateways”:

– entry into post graduate training

– at the time of primary licensure

“Gateways” are viewed as licensure decision points, not examination events

Themes continued

• Patient-centered

• Competency based

• Meaningful integration of the scientific foundations of medicine in all exam components

• Support valid secondary uses if this does not detract from primary mission of examination

Within each Gateway

• Exam components are non-compensating– Separate hurdles for scientific foundations,

knowledge required for practice, and clinical skills

• To the greatest extent possible, individual test items in each component will integrate relevant basic science and clinical materials

• Increased emphasis on information gathering, interpretation of literature and evidence-based medicine

Scientific Foundations of Medicine

• We anticipate building assessment instruments that measure not only retention of current material, but also the ability to evaluate new information and apply scientific principles to solve clinical problems

• Both Gateways would contain these materials

One Hypothetical Exam Model*

• Gateway A– Scientific Foundations of Medicine– Knowledge Required for Supervised Clinical

Practice– SP-based Clinical Skills

• Gateway B– Scientific Foundations of Medicine– Knowledge Required for Unsupervised Practice– Computer based Clinical Skills

* PVS 2.05.08

If we are successful, it will be difficult to classify individual items into pure basic

science or clinical medicine areas

We will probably wind up with a “mostly” Scientific Foundations section, and a “mostly” Knowledge for Clinical Practice section in at

each Gateway

CEUP recommendations are currently in draft stage

These are high level recommendations that will enable the process and set the guidelines

for continued development and stakeholder review

CEUP will likely not specify the nature, number or timing of exam events required

for each gateway

1. Competency based assessment

• At entry into graduate training, doctors must have minimum competencies necessary to safely care for patients.

• At time of licensure, a higher level of these competencies, together with others acquired during GME, are necessary.

• Competencies that can be measured in valid, reliable, & practical manner, should be incorporated into the USMLE.

2. Two Patient-Centered decision points

1. Entry into Post-graduate training

2. Primary Licensure

3. Integrate the scientific foundations of medicine into all components

• The basic sciences are essential components of good medical practice

• Current separation of Basic Science and Clinical Knowledge is artificial

• To the greatest extent possible, individual test items in each component should integrate relevant basic science and clinical materials

4. Continue to support the educational process

• Current Steps 1 and 2 are used by many schools for curriculum evaluation and for promotion and graduation decisions

• Assure that valid, reliable, and secure assessment instruments are available to schools that wish to use them

5. Carefully study issues surrounding score reporting

Take the time necessary to do this right

6. Allow evolution

• USMLE must be viewed in context of lifelong learning & competency assessment.

• There may be new essential medical competencies that will arise

• USMLE design must be sufficiently flexible to incorporate these to the extent possible.

Next steps

• Development of a series of “straw” models for assessment instruments

• Prototype test committee meetings

• Reality testing of straw models with stakeholders

• Study of practical implications of models

• On-going study of scoring issues

• Feasibility report to governance

Format and item development, review, pretesting begins

March 2008

Committee work begins

Fall, 2009

Sample materials, content outlines posted

July 2012-July 2014

“Live” exams become available

Infrastructure systems developed

Scoring and reporting models developed

Supporting materials developed

July 2011

“Practice” tests become available

Fall, 2010

Class of 2013 begins studies

aamc activities amsndc cas representation

Documents

aamc cas liaison

aamc symposium

aamc task force

aamc executive committee

aamc executive council

aamc cas spring meeting

hhmiaamc workgroup

hhmiaamc panel