evidence for the effectivenessof - podiatry for the...enee, or inconclusive. statisticallysig¬...

Evidence for the Effectiveness of CMEA Review of 50 Randomized Controlled TrialsDavid A. Davis, MD; Mary Ann Thomson, BHSc (PT); Andrew D. Oxman, MD; R. Brian Haynes, MD, PhD

Objective.\p=m-\Toassess the impact of diverse continuing medical education(CME) interventions on physician performance and health care outcomes.Data Sources.\p=m-\Usingcontinuing medical education and related phrases, we

performed regular searches of the indexed literature (MEDLINE, Social ScienceIndex, the National Technical Information Service, and Educational Research In-formation Clearinghouse) from 1975 through 1991. In addition, for these years, weused manual searches, key informants, and requests to authors to locate other in-dexed articles and the nonindexed literature of adult and continuing professionaleducation.Study Selection.\p=m-\Fromthe resulting database we selected studies that met the

following criteria: randomized controlled trials; educational programs, activities, orother interventions; studies that included 50% or more physicians; follow-upassessments of at least 75% of study subjects; and objective assessments of ei-ther physician performance or health care outcomes.Data Extraction.\p=m-\Studieswere reviewed for data related to physician specialty

and setting. Continuing medical education interventions were classified by theirmode(s) of activity as being predisposing, enabling, or facilitating. Using the statis-tical tests supplied by the original investigators, physician performance outcomesand patient outcomes were classified as positive, negative, or inconclusive.Data Synthesis.\p=m-\Welocated 777 CME studies, of which 50 met all criteria.

Thirty-two of these analyzed physician performance; seven evaluated patient out-comes; 11 examined both measures. The majority of the 43 studies of physicianperformance showed positive results in some important measures of resource uti-lization, counseling strategies, and preventive medicine. Of the 18 studies of healthcare outcomes, eight demonstrated positive changes in patients' health care out-comes.

Conclusion.\p=m-\Broadlydefined CME interventions using practice-enabling or

reinforcing strategies consistently improve physician performance and, in some in-stances, health care outcomes.

(JAMA 1992;268:1111-1117)

FOR SEVERAL decades, recognitionof the importance of continuingmedicaleducation (CME) has been dogged by

From the Departments of Family Medicine (Drs Davisand Oxman), Clinical Epidemiology and Biostatistics(Drs Oxman and Haynes), and Medicine (Dr Haynes);and the School of Occupational Therapy and Physio-therapy (Ms Thomson), Faculty of Health Sciences,McMaster University, Hamilton, Ontario, Canada.Reprint requests to Continuing Health Sciences Ed-

ucation, Faculty of Health Sciences, 1200 Main St West,Chedoke Campus, Bldg 74, Hamilton, Ontario, CanadaL8N 3Z5 (Dr Davis).

questions regarding its effects on phy¬sician competence and performance andwhether it improves health care out¬comes.1 As the body of biomedicai lit¬erature grows, as the CME "industry"grows, and as regulatory bodies con¬sider the nature of competence and itsrelationship to CME, the need acceler¬ates for a careful, critical appraisal ofthe impact of this longest and arguablymost important phase of medical edu¬cation. To meet that need, we examined

the strongest evidence available on thenature and impact of CME.Critical reviews of the CME litera¬

ture are not new. Over a decade ago,Stein2 analyzed those features ofCME methods that changed physicianbehaviors. Other reviews followed,3·4including one reported by two of us(R.B.H. and D.A.D.) in 1984.5 Theseanalyses provide strong evidence thatsome CME interventions effectchanges in competency (the physi¬cian's ability to perform in the testsituation), less strong evidence forchanges in actual clinical performance,and only weak demonstration ofchanges in health care outcomes. Tworecent reviews have provided new

perspectives: McLaughlin and Donald¬son's6 categorization of the educa¬tional intervention, and Beaudry's7application of meta-analytic tech¬niques in evaluating CME. The impe¬tus for this review was the substan¬tial increase in the number and rigorof CME studies over the past fewyears and the growing interest inCME as a practice-based discipline.8Since 1979, we have maintained a com¬

prehensive collection ofCME literature.Initially developed in print format,9 theproject's scope has broadened, and thecurrent computerized bibliography10 con¬tains over 1500 citations relevant toCME, broadly defined herein as all waysby which physician learning and clinicalpractice may be altered by educationalor persuasive means.In this article, we review a subset of

the studies comprising that bibliogra¬phy, ie, randomized controlled trials thatstudy the effect of specific CME inter¬ventions on either physician perfor¬mance or health care outcomes. Our aimwas to identify attributes of such inter¬ventions that most affect physician per¬formance and health care outcomes and

Downloaded From: http://jamanetwork.com/pdfaccess.ashx?url=/data/journals/jama/9641/ by a SCELC - Western University of Hlth Sci User on 03/02/2017

Types of Continuing Medical Education Interventions and Their OutcomesIntervention Type

Predisposing*Predisposingand Enablingf

Predisposingand Reinforcing^

PhysicianPerformance

Health Careor PatientOutcomes


Health Careor PatientOutcomes


IHealth Careor PatientOutcomes

All Three Categories orMultipotential Interventions!

I IHealth Care

Physician or PatientPerformance Outcomes

No. of interventions withpositive results

No. of Interventions withnegative or inconclusiveresults 1

*Academlc detail visits; computer-generated Information; consultations; didactic presentations, lectures; knowledge testing; needs assessment; printed materials; traineeship;workshops; tutorials and smali groups.fCllnlcal policy and practice guidelines; patient education materials or Instruction; information from patients; practice protocols, algorithms, and flow charts.^Feedback and reminders.§Chart reviews and chart-stimulated recall; influential educational leaders and clinical opinion leaders.

to derive conclusions for further prac¬tice and research.

METHODSData SourcesOur search of relevant CME litera¬

ture included studies published from1977 through 1991 and comprised thefollowing steps:

1. Monthly selective dissemination ofinformation printouts produced by theNational Library ofMedicine, Bethesda,Md (MEDLINE), using relevant keywords and Medical Subject Headings(MeSH) including medical education,continuing medical education, CME,continuing education, peer review, qual¬ity assurance, chart audit, medical au¬dit, physician competence, performanceassessment, health services research, uti¬lization review, adult education, andcontinuing professional education. Allarticles using any of these terms werereviewed.2. Biannual searches of other data¬

bases, including the Social Science In¬dex, the National Technical InformationService, and the Educational ResearchInformation Clearinghouse, using theabove-named key words.3. Manual searches of the reference

lists of articles.4. Requests of key informants in the

supporting organizations for nonindexedarticles, including the Journal of Con¬tinuing Education in the Health Pro¬fessions and journals in the field ofadulteducation.5. Requests to the correspondent of

each identified paper to locate additionalstudies that met our stated criteria.

Selection of ArticlesTo determine as rigorously as possi¬

ble the impact of CME interventions onthe performance of physicians in prac¬tice and on the status of their patients,articles were selected if they met thefollowing criteria.

Design.—Individual physicians orfunctional units of physicians (eg, hos¬pitals) randomly allocated to one or moreintervention group(s) and a controlgroup.Intervention.—A replicable descrip¬

tion of any educational activity or pro¬gram intended to improve physician per¬formance or patient outcomes. An in¬terventionwas considered "educational"if it primarily consisted of the transferof information to targeted physicians andwas intended to affect physician perfor¬mance through persuasion; interventionsthat used official sanctions or rewards,such as financial incentives, or that pre¬sented already available clinical infor¬mation, such as laboratory test results,in a different format without new infor¬mation were excluded.Subjects.—Administrative units in

which physicians formed at least 50% ofthose observed. Studies of residentswere included, as the residency experi¬ence is similar to the "traineeship" de¬scribed in the CME literature, and manypracticing physicians work in similarteaching-hospital or clinic settings.Outcomes.—Evaluation of an inter¬

vention's effect by objectively observedphysician performance (behavior) or pa¬tient outcomes.Follow-up.—Assessment ofoutcomes

for at least 75% of study participants.Data Analysis.—Provision of suffi¬

cient data for statistical analysis or in¬dication of statistical significance.To appraise the evolution of knowl¬

edge in this field, these criteria followedthe pattern of article selection in our1984 review of the literature.5

Data ExtractionOne of us (M.A.T.) extracted data in

each ofthe following areas: study setting,type ofphysicians targeted by the study,clinical area of practice and characteris¬tics of the CME intervention, and effectson physician performance and patient

outcomes. At least one of the other au¬thors checked the data for accuracy.

Classification of InterventionsFor this review, we grouped CME

interventional strategies into four types,using the classification ofGreen and col¬leagues11 of factors relevant to behaviorchange in health promotion (Table).These factors,modified for our purposes,included predisposing (communicatingor disseminating information), enabling(facilitating the desired change in thepractice site), and reinforcing (by re¬minders or feedback). The four inter¬vention types thus derived included: type1, using predisposing factors only; type2, predisposing plus enabling; type 3,predisposing plus reinforcing; and type4, a single multifaceted maneuver, suchas chart review with a peer physician,or a combination of interventions span¬ning all three types.AnalysisQuantitative data regarding the ef¬

fects of CME on physician performanceand on patient outcomeswere extractedand tabulated. Although many of thestudies reported multiple analyses, weexamined only the primary measures ofphysician performance and patient out¬comes for each study. Since the out¬comes, physicians targeted, and typeof intervention differed substantiallyacross studies, we did not calculate the"average effect" of CME. For similarreasons, only weak inferences could bebased on between-study comparisons ofthe relative effectiveness of differentinterventions. This overview thereforepresents a qualitative analysis of results,reiving in large part on tests of statis¬tical significance reported in the trialsand the original investigators' assess¬ments of the practical importance of ob¬served differences.Study results were classified as ei¬

ther statistically significant, no differ-


enee, or inconclusive. Statistically sig¬nificant "positive" studies were thosethat found a statistically significant dif¬ference (at least P<.05) between theintervention group and a control groupthat received no education or a less po¬tent CME activity. All the results clas¬sified as statistically significant wereconsidered clinically or educationally im¬portant by the original investigators.No-difference, or "negative" studies

were those that did not find a statisti¬cally significant difference betweenstudy and control groups and possessedsufficient power to rule out what inves¬tigators defined as a practically impor¬tant difference. This category includedstudies in which the 95% confidence in¬terval excluded any difference in favorof the experimental CME intervention.Inconclusive studies were those that

found no difference or a trend toward adifference, but may have lacked suffi¬cient power to demonstrate or excludea statistically significant or practicallyimportant difference.RESULTSOf the 1445 CME articles in the CME

literature database, 777 described in¬terventions; of these, 86 were random¬ized controlled trials, of which 50 metthe criteria outlined in the "Methods"section.12"61

Descriptive Analysis of the StudiesThe majority (45) of the studies ana¬

lyzed the performance of internists,general practitioners, and family physi¬cians, or their patient outcomes. Alsodescribed were studies of obstetricians(two), emergency department physi¬cians (one), and pediatricians (two).Several studies described only primaryphysicians, or "MDs." Sixteen studiesobserved physicians in private practiceand their patients; 16 analyzed inter¬ventions in hospital outpatient clinics;four, health maintenance organizations;and one, a family practice teaching unit.Thirteen studies focused on inpatienthospital settings; 15 included residentsor house staff; seven, a mixture of res¬idents and supervising faculty; andeight, residents alone.Thirty-two trials analyzed physician

performance, seven studies evaluatedpatient outcomes, and 11 examined bothmeasures. The clinical aspects of theCME methods were of several types:comprehensive clinical management ofgeneral medical conditions, includinginvestigation, diagnosis, and treatment(22 studies); use of laboratory and ra¬

diologie investigations (11); prescribingpractices (six); patient counseling (six);and primary prevention activities(five).

Physician Performance StudiesGeneral ClinicalManagement.—Six¬

teen studies focused on physician per¬formance in general clinical manage¬ment, and 12 reported postinterventionchanges in physician performance.* Fourof these31,50"52 used opinion leaders or "ed¬ucational influentials" to modify physi¬cian behavior; others used well-designedinstructional methods28,57 or computer-generated reminders46 toeffect changes.Despite the overall change in the de¬sired direction reported in these stud¬ies,* not all the results measuredachieved statistical significance. For ex¬ample, Putnam and Curry,43 assessingthe impact of a criteria-setting work¬shop in family medicine, reported im¬provement in only two of fourmeasures.Four studies16·24·25·49 failed to demonstratepositive changes: twowere negative,24·49and two were inconclusive.16,25Investigations.—Eleven studies as¬

sessed the impact of CME training us¬

ing computerized feedback on the use orcost of laboratory tests and roentgeno-grams.f Of these, only one61 failed toshow a difference between experimen¬tal and control groups, and that studymay have lacked sufficient power to dem¬onstrate any reduction. Ten studies dem¬onstrated positive changes by using acombination of intervention measures.For example, Chassin and McCue,14 us¬ing didactic sessions, printed material,and feedback, altered pelvimetry ratesin an obstetrical setting. Similarly, Ever¬ett and colleagues22 reported that chartreview by a clinical supervisor reducedthe frequency and cost of laboratory testorders. One study showed an increase inthe appropriate use of roentgenograms(barium enemas) and sigmoidoscopy byworkshop-trained physicians.42Prescribing Practices.—Six studies

analyzed the effects of a variety of in¬terventions on physicians' prescribinghabits; five were positive.12·23,26,37,40 Forexample,Avorn and Soumerai12 and Mc-Connell et al37 coupled visits by trainedphysician-educators such as pharmacists("academic detail persons")with printedmaterial and feedback to lower the pre¬scribing of targeted drugs and showedthat simple mailed materials were inef¬fective by themselves. Meyer and asso¬ciates,40 studying the effects on poly-pharmacy of a newsletter or extensivesupervision including an opinion leader,feedback, and chart review, reportedthat both were equally effective in ac¬

complishing performance objectives.Gehlbach et al,23 using computerizedfeedback, reported a significant increase

*References 28,30-32,36,43,46,48,50-52,57.\s=d\References13-15,20,24,35,42,54,55,61.

in generic drug prescribing in a familypractice teaching unit. In contrast, Her-shey et al26 demonstrated that comput¬erized feedback could significantly re¬duce prescription costs, but only at theconclusion of the study, and then only intwo offive measures. A follow-up studyby Hershey et al,27 using computerizedfeedback plus a newsletter, produced nochange compared with computerizedfeedback alone.Counseling Strategies.—Interven¬

tions designed to assist physicians inpatient counselingwere reported in fivestudies,18'19-29'33·69 four ofwhich dealt withsmoking cessation18,19,29·59 and one withcompliance-enhancing behaviors in pe¬diatrics.33 Each of the smoking cessa¬tion studies used complex educationalinterventions, such as didactic presen¬tations, practice-oriented workshops,printed material, reminders, and patienteducation materials to improve physi¬cian behavior. Similarly, Maiman et al33demonstrated that print material, cou¬pled with didactic and problem-basedsessions, could improve pediatricians'compliance-enhancing skills.PreventiveCare.—All five studies fo¬

cusing on the ability of CME to improveprimary prevention practices showed im¬provement in at least some major out¬comes. Three of the studies used com¬

puter-generated reminders to physiciansto perform certain tasks: McDonald etal38 noted improvement in 14 of 15 mea¬sures; McPhee et al,39 adding print andpatient education strategies, found im¬provement in eight of 11 measures; andTierney et al53 determined that remind¬ers and feedback, used separately, pos¬sessed as much influence on preventivecare as when both were used together.Two studies used a combination of di¬dactic presentation, feedback, andprinted materials. Palmer et al41 foundsignificant improvement in only two ofeight measures spanning awide varietyof screening maneuvers; Winickoff andcolleagues60 effected changes in colo-rectal cancer screening.Other aspects of clinical practice not

assessed in these 50 randomized con¬trolled trials include the clinical reason¬ingprocess (except in the studybyHealeand associates25), procedural or techni¬cal skills (except in the study of sig-moidoscopy by Perera et al42), the re¬ferral process, and information manage¬ment skills.Many clinical areas and someentire disciplines, such as surgery, werenot represented in these studies.Health Care Outcome StudiesEighteen studies analyzed the effects

of CME interventions on patient orhealth care outcomes; 10 had negativeor inconclusive results; eight demon-


strated some positive changes in at leastone major measure. Studies with neg¬ative results included those focusing onsmoking cessation,18·19·29 hypertensioncontrol,21·24·44 and the functional status ofpatients.47 Three studies, in generalmed¬ical care,48 asthma,58 and arthritis,52 gen¬erated inconclusive patient outcomes.We found eight studies with positive

patient care outcomes in at least onemajor measure. In smoking cessation,both Cohen and associates17 and Wilsonet al59 increased patient quit rates aftercounseling by CME-trained physiciansfor at least 3 months postintervention.The remaining studies used a variety ofCME interventions to alter patient out¬comes. Lomas et al31 found that clinicalopinion leaders (local clinicians able toinfluence their colleagues' performanceby exemplary practice, role modeling,consultation, and teaching) increased theproportion of vaginal births after cesar-ean sections; Maiman and associates33employed a mixture of workshop, di¬dactic presentation, and printed mate¬rials to improve patient compliance; andVinicor et al56 reported that didactic pre¬sentations, protocols, reminders, and pa¬tient education strategies improved di¬abetic outcomes. Restuccia45 determinedthat feedback from a coordinator, occa¬sionally augmented by that of a staffphysician, reduced inappropriate patienthospital days. Linn,30 in an emergencydepartment setting, demonstrated thata combination ofprotocols, feedback, andtraditional teachingmethods altered pa¬tient outcomes, but in only one of fourmeasures in the emergency departmentand in one of eight measures for thosepatients admitted. Finally, Rogers andassociates46 used computerized medicalrecords to prompt physician activities,resulting in some positive patient out¬comes in mitigating obesity and renaldisease, though not in reducing hyper¬tension.

Methods of InterventionThe 50 studies used a total of 74 dis¬

crete CME methods, all of which, byvirtue of their educational intent, con¬tained some element of information dis¬semination. Of these 74,15 were type 1(predisposing only); 12, type 2 (predis¬posing and enabling); 31, type 3 (pre¬disposing and reinforcing); and 16, type4 (multipotential interventions or amix¬ture of all three methods) (Table).Type 1 predisposing strategies pro¬

duced mostly negative or inconclusiveresults: while seven out of eleven dis¬played positive physician performancechanges, only one of the six attempts tochange health care status accomplishedthis objective.Most studies using printedmaterialswere negative12·21·44·49 or incon-

elusive16; however, in two studies,printed materials were noted to be aseffective in altering prescribing prac¬tices or use of investigations as werewere chart review, feedback, and infor¬mation derived from patients.35,40Work¬shops or small-group discussions inwhich no explicit attempt was made tofacilitate physician performance by in-trasession practice strategies were in¬conclusive.25,58 In contrast, studies usinglearning experiences that incorporatedknowledge testing and practice needsassessment strategies did succeed in im¬proving some aspects of physician per¬formance.28,67Of the 12 type 2 predisposing and en¬

abling activities, 10 attempted to changephysician performance and six measuredhealth care outcomes. Nine of the formerwere positive,* as were two of six in thelatter studies.33,56Patient education tech¬niques were effective in altering physi¬cian performance in studies that cou¬pled these methods with academic de¬tail visits,12 printed materials,29 orwork¬shops,29 or on health care outcomes whenused alone.12 Information about specificpatients, derived from health statusquestionnaires or interviews, enhancedphysician performance in one study,32but had no patient effects in another.47The dissemination of clinical policies orpractice guidelines alone showed a neg¬ative effect.31 However, more specificpractice protocols or clinical algorithms,coupledwith printedmaterials and work¬shops, changed physician performancein family practice43 and, with feedback,in the emergency department setting.30Other studies using workshops rein¬forced by didactic presentations and"practice rehearsal" strategies improvedphysician performance33 and, in one in¬stance, patient care outcomes.33 Whiteand associates58 studied a small groupthat generated practice guidelines, butthese failed to demonstrate an effect onhealth care outcomes. Finally, computer-generated information used by Tierneyand associates54,66 in two studies alteredphysician use of laboratory investiga¬tions.Type 3 methods consisted primarily

of feedback and/or reminders in conjunc¬tion with predisposing methods. Feed¬back was used in 23 of the 31 interven¬tions studied,t and effected a positiveperformance change in 12 instances whenused alone13,16,23,26,36,46,53,61 or in conjunc¬tion with didactic presentations, work¬shops, academic detail visits, or printedmaterials.14,24,27,36,37,41,60 Two studies24,31demonstrated that performance feed-

*References 12,29,32,33,42,43,54,55.\s=d\References13-15,23,24,26,27,31,35,37,41,45,53,

60,61.

back induced no change in patient out¬comes, whereas one study45 that gavephysicians feedback about patients'lengths of stay improved patient out¬comes. Practice reminders, also com¬

monly used, had a positive effect on phy¬sician performance36·38·46·53 and patientoutcomes.17·46Type 4 methods, using a combination

of all three CME strategies, comprisedthe majority of this category (10 of 16interventions described)^ and producedpositive results in all eight assessmentsof physician performance, and in four ofseven analyses of health care status. Inaddition, there were two multifacetedsingle strategies within this category:opinion leaders and chart review. Theeffectiveness of educational leaders orinfluential persons on behavior andhealth care outcomes was demonstratedby Lomas and colleagues.31 Stross andassociates5052 established the perfor¬mance-altering potential of teachers andeducational leaders in the managementof arthritis and respiratory disease,though one study52 related to patients'health status provided inconclusive re¬sults. Finally, chart review, in which afacultymember or supervisor discussedpatient records with a resident, im¬proved physician performance in threestudies.22·34·40COMMENTOur study located 777 articles testing

CME interventions, of which 50 wererandomized controlled trials assessingphysician performance and/or healthcare outcomes. This more than seven¬fold increase over the seven randomizedcontrolled trials described by us in 19845and the almost tripling of the denomi¬nator over this period illustrates a sub¬stantial increase in research by thoseinterested in enhancing physician per¬formance. In addition, we note a widerange of rigorously tested interventionsthat substantially broadens the tradi¬tional definition of continuing medicaleducation. Interventions not describedby us in 1984 included individualizedchart reviews, computer-generated in¬formation, and complex, role-reinforc¬ing activities in workshops.While the randomized controlled trial

is indispensable formethodological rigor,we recognize several difficulties withexecuting such studies.5,62Although gen¬erally well designed, many studies didnot provide enough details about theirphysician populations37,45,48 orblind theirassessors to the CME group. Becauserandomized controlled trials are diffi¬cult and expensive to perform and usu-

\s=dd\References18,19,30,36,39,40,48,56,59.


ally require volunteer physicians, thegeneralizability of our findings may belimited to physicians who are above theirpeers in performance and thus willing tohave their performance and their pa¬tients health status scrutinized.Although these studies eliminate

many concerns about contamination andcointervention, they frequently lackedqualitative details63 that would help inthe delineation ofphysician managementpatterns, learning processes, or forcesfor and impediments to change.64 In lightof the variance in the types of clinicians,clinical practices, settings, outcome mea¬sures, and CME strategies, as well asother aspects of the study designs, com¬paring the effectiveness of differentCME strategies should be viewed as

hypothesis-generating rather than hy¬pothesis-testing.65 In addition, becauseof wide variations in the way outcomeswere reported in these studies, we fo¬cused on tests of statistical significancerather than on the magnitude of effect.Although it appears reasonable to as¬sume that all of the statistically signif¬icant studies found differences thatwereclinically important,66 it is not possibleto assume that the inconclusive studiesruled out a clinically important effect.67Thus, our conclusions about interven¬tions that appear to be effective are

stronger than our conclusions aboutthose that do not.We attempted to correct for publica¬

tion bias that prevents negative or in¬conclusive studies from being submit¬ted for publication, but such bias maystill exist. Therefore,while the strengthof inferences that can be drawn fromour overview of the relative effective¬ness of different CME strategies is lim¬ited, these inferences reflect the presentstate ofknowledge in the CME field andmay lead to recommendations for prac¬tice improvements and future research.Physician PerformanceIf physician performance is consid¬

ered a benefit ofCME, these studies areencouraging with regard to short-term,post-CME interventions and lend them¬selves to discussion ofwhat specific strat¬egies and activities appear to work andin what circumstances.These studies and the relevant liter¬

ature68 suggest that clinicalmanagementand patient counselingbehaviors are rel¬atively complex and more difficult tochange. Prescribing and preventivepractices and use of investigations ap¬pear easier to alter. Nearly one third ofCME trials attempting to measure

change in the complex behaviors of gen¬eral clinical management—incorporat¬ing investigation, diagnosis, and man¬

agement69—did not succeed; even among

those trials considered "positive," notall themeasures achieved statistical sig¬nificance.The reasons for failure may include

attempts to change clinical phenomenasuch as hypertension outcomes in a rel¬atively well-controlled population,24 orthe use of inadequate, single-factor in¬terventions.21,49 More potent practice-linked interventions, such as experiencedclinical leaders and computer-generatedreminders, were effective. Traditionalworkshops or didactic presentations,when augmented with objective ways ofassessing individual physician's require¬ments (such as chart audits and knowl¬edge testing) appear to establish base¬lines for affecting real outcomechange.28,57Health counseling appears to be al¬

tered throughmore complex strategies70;especially important were explicit in¬structions,33 opportunities to rehearsesuch strategieswith simulated patients,59videotapes, and role playing.18,19 Simplerbehaviors were more consistently al¬tered in these trials; for example, all butone61 of the 11 studies of test ordering*demonstrated positive change. Practice-linked maneuvers such as reminders,feedback, computer-available informa¬tion,54·55 and chart review by supervi¬sors or faculty members22,34 appeared tobe effective change agents, as did theuse of a "probabilities curriculum" re¬inforced by discussions on the ward.20For prescribing practices, in whichgvei2,23,26,37,4o 0f sjx studies27 showed pos¬itive change, having academic detail per¬sons giving on-site information and feed¬back appears to be helpful and cost-effective,71 while printed materials ap¬pear less effective.12Finally, in disease prevention, remind¬

ers and feedback, used concurrently, areeffective in improving physician perfor¬mance.72Health Care Outcome StudiesThe increase since our previous re¬

view5 in rigorously designed CME stud¬ies that analyze health care status andpatient outcomes provides a more op¬timistic picture of the impact of CME.Of note are the smoking cessation stud¬ies,17"19,29,59 of which two show positiveresults in patients at least severalmonths postintervention.17·59 Addition¬ally, studies by Lomas et al,31 Maiman etal,33 Restuccia,45 and Vinicor et al56 con¬firm that at least some changes in pa¬tient outcomes are possible. Explana¬tions for "negative" studies—still themajority ofthe studies in this category—include lack of a direct relationship be¬tween the CME-induced change in phy-*References 13-15,20,22,24,35,42,54,55,61.

sician performance and improved patientoutcomes and such impediments to op¬timal outcomes as noncompliance withmedication. Further, a "ceiling effect"may occur in studies such as those ofhypertension,21,24,44 in which physiciansand patients have limited ability to im¬prove their performance because theyare already functioning at or near theirmaximum capabilities.Successful and UnsuccessfulInterventionsThere appears to be a direct relation¬

ship between the intensity of the inter¬vention and the number of studies withpositive outcomes. In these studies, in¬terventions using only predisposing el¬ements to disseminate informationwereless apt to induce physician performancechange and registered little or no effecton health care outcomes. Much of thecriticism aimed at methods frequentlyused in the formal programs that arethe domain of traditional CME provid¬erswould seem justified.44,49 In contrast,those studies that used enabling and/orreinforcing elements were more effec¬tive in changing outcomes; for example,workshops that provide more opportu¬nity for case discussion and rehearsal ofpractice behaviors are considerablymoreeffective than are more didactic pro¬grams. These studies also support oneof the major tenets of adult education73:that the objective determination ofprac¬tice or learning needs is a necessaryprerequisite for effective education.Two further methods in the type 1

category merit comment. Althoughprinted materials alone demonstrate a

relatively weak effect on physician per¬formance, theymay be among the manyfactors affecting performance change.64From the results of at least two stud¬ies,26,58 small-group learning as a singleentity has failed to demonstrate its abil¬ity to change physician performance25or patient outcomes,58 possibly owing tothe inadequate potency of small-grouplearning in the CME setting, or to un¬directed discussion. Given the increas¬ing use of small-group learning in med¬ical schools and the theoretical basis ofthe process,69 we urge further researchin this area. Finally, activities describedas "academic detail visits" appear to beeffective change agents and worthy offurther study, both in drug-prescribingand in other clinical domains.From our review of type 2 CME, two

strategies that appear to facilitate prac¬tice change are patient education andcomputerized practice-based informa¬tion. Practice guidelines, when usedalone, were not effective. Type 3 CME,which includes feedback and reminders,appears to overcome many of the logis-


tical and sociological barriers to facili¬tating optimal physician performance.74Finally, type 4 interventions demon¬strate the ability to change outcomeswith a host of strategies derived fromall three categories, such as studies ofsmoking cessation and diabetes manage¬ment17"19·29'56·59 that used a mixture ofworkshops, patient educationmaterials,printed materials, and reminders. Vini-cor et al56 demonstrated that patient ed¬ucation materials can alter patient out¬comes either singly or, with greater im¬pact, in conjunction with other CMEstrategies. Chart review or chart-stim¬ulated recall appears to be an effectiveand comprehensive CME activity, in¬corporating elements of informationtransfer, competency assessment, re¬minders ofdesirable practices, feedbackto the physician, and the opportunityfor performance-enabling sugges¬tions.75"77 Similarly, the example of ed¬ucational or opinion leaders—derivedfrom research on the "educational in¬fluential" in community hospitals78—ap¬pears to act in a comprehensive and ef¬fective manner.

Implications for Researchand PracticeThis review indicates the need to ad¬

dress several important issues in CME.First, there are some noticeable gaps inthe kinds of CME interventions stud¬ied, particularly when compared withphysicians' stated CME preferences.79,80Journal reading, for example—clearly apopular mode of learning for most phy¬sicians81—to our knowledge was notstudied to any extent and certainly notby controlled trials.Second, promising interventions de¬

serve further testing and use—partic¬ularly chart review, education leaders,academic detail visits, and multiphasicintervention activities. Third, the clin¬ical outcomes chosen in most CME stud¬ies appear to be determined somewhatmore by their ease ofmeasurement (eg,blood pressure) or by their cost (eg, in¬vestigations) than by their clinical im¬perative. In this regard, we suggest thatCME researchers develop a researchagenda that would include studies eval¬uating the effects of CME interventionson major causes ofmorbidity and mor¬tality. Fourth, both providers and con¬sumers of CME need to be aware of theclear lessons derived from this review:that CME is more effective when it in¬corporates practice-based enabling andreinforcing strategies and that adequateassessment ofphysicians' needs leads toincreased potential for change.Finally, these randomized controlled

trials provide new evidence supportingthe effectiveness ofbroadly defined, com-

plex, practice-linked CME. Based on thisevidence, we urge the development ofan improved evidence-based CME de¬livery system and further research intothis last and most complex arena of phy¬sician education.

The authors acknowledge the financial support ofthe Research and Development Resource Base inContinuing Medical Education (McMaster Univer¬sity, Hamilton, Ontario, Canada) in facilitating thewriting of this review; the American MedicalAssociation (Chicago, 111); the Canadian MedicalAssociation (Ottawa, Ontario, Canada); the SocietyofMedical College Directors ofContinuingMedicalEducation (Chicago, 111); the Royal College of Phy¬sicians and Surgeons of Canada (Ottawa, Ontario,Canada); and the Alliance for Continuing MedicalEducation (Lake Bluff, 111).Dr Oxman is a career scientist of the Ministry of

Health, Ontario, Canada. Dr Haynes is a recipientof the National Health Scientist Award, fromHealth and Welfare Canada.

References1. Miller GE. Continuing education for what? JMed Educ. 1976;42:320-326.2. Stein LS. The effectiveness of continuing med-ical education: eight research reports. JMed Educ.1980;56:103-110.3. Lloyd JS, Abrahamson S. Effectiveness of con-tinuingmedical education, a review ofthe evidence.Eval Health Professions. 1979;2:251-280.4. Bertram DA, Brooks-Bertram PA. The evalua-tion of continuing medical education: a literaturereview. Health Educ Monogr. 1977;5:330-362.5. Haynes RB, Davis DA, McKibbon A, Tugwell P.A critical appraisal of the efficacy of continuingmedical education. JAMA. 1984;251:61-64.6. McLaughlin PJ, Donaldson JF. Education of con-tinuing medical education programs: selected lit-erature, 1984-1988. JContinuing Educ Health Pro-fessions. 1991;11:65-84.7. Beaudry JS. The effectiveness ofcontinuingmed-ical education: a qualitative synthesis. J Continu-ing Educ Health Professions. 1989;9:285-307.8. Watts MSM. On the third segment of healthprofessions education. J Continuing Educ HealthProfessions. 1989;9:129-130.9. Chambers L, Davis D, Dok C, et al. The Impactof CME: An Annotated Bibliography. Hamilton,Ontario: McMaster University Press; 1979.10. Annual Report of the Research and Develop-ment Resource Base in CME. Hamilton, Ontario,Canada: McMaster University Faculty of HealthSciences; 1991.11. Green L, Kreuter M, Deeds S, Partridge K.Health Education Planning: A Diagnostic Ap-proach. Palo Alto, Calif: Mayfield Press; 1980.12. Avorn J, Soumerai SS. Improving drug ther-apy decisions through educational outreach. NEnglJ Med. 1983;308:1457-1463.13. Berwick DM, Coltin KL. Feedback reduces testuse in a health maintenance organization. JAMA.1986;255:1450-1454.14. Chassin MR, McCue SM. A randomized trial ofmedical quality assurance. JAMA. 1986;256:1012\x=req-\1016.15. Cohen DI, Jones P, Littenberg B, NeuhauserD. Does cost information availability reduce phy-sician test usage? Med Care. 1982;20:286-292.16. Cohen SJ, Weinberger M, Hui SL, TierneyWM,McDonald CJ. The impact of reading on physicians'nonadherence to recommended standards of med-ical care. Soc Sci Med. 1985;21:909-914.17. Cohen SJ, Stookey GK, Katz BP, Drook CA,Smith DM. Encouraging primary care physiciansto help smokers quit. Ann Intern Med. 1989;110:648-652.18. Cummings SR, Richard RJ, Duncan CL, et al.Training physicians about smoking cessation: a con-trolled trial in private practices. J Gen Intern Med.1989;4:482-489.

19. Cummings SR, Coates TJ, Richard RJ, et al.Training physicians in counseling about smokingcessation: a randomized trial of the 'Quit for Life'program. Ann Intern Med. 1989;110:640-647.20. Davidoff F, Goodspeed R, Clive J. Changingtest ordering behavior: a randomized controlledtrial comparing probabilistic reasoning with cost\x=req-\containment education. Med Care. 1989;27:45-58.21. Evans CE, Haynes RB, Birkett NJ, et al. Doesamailed continuing program improve physician per-formance? results of a randomized trial in antihy-pertensive care. JAMA. 1986;255:501-504.22. Everett GD, deBlois S, Chang P-F, Holets T.Effect of cost education, cost audits, and facultychart review on the use of laboratory services. ArchIntern Med. 1983;143:942-944.23. Gehlbach SH, Wilkinson WE, Hammond WE,et al. Improving drug prescribing in a primary carepractice. Med Care. 1984;22:193-201.24. Gullion DS, Tschann JM, Adamson TE, CoatesTJ. Management of hypertension in private prac-tices: a randomized controlled trial in continuingmedical education. J Continuing Educ Health Pro-fessions. 1988;8:239-255.25. Heale J, Davis D, Norman G, Woodoward C,Neufeld V. A randomized controlled trial assessingthe impact of problem-based versus didactic teach-ing methods in CME. Proc Conference Res MedEduc. 1988;27:72-77.26. Hershey CO, Porter DK, Breslau D, Cohen DI.Influence of simple computerized feedback on pre-scription charges in an ambulatory clinic. Med Care.1986;24:472-481.27. Hershey CO, Goldberg HI, Cohen DI. The ef-fect ofcomputerized feedback coupled with a news-letter upon outpatient prescribing charges. MedCare. 1988;26:88-93.28. Jennett PA, Laxdal OE, Hayton RC, et al. Theeffects of continuing medical education on familydoctor performance in office practice: a randomizedcontrol study. Med Educ. 1988;22:139-145.29. Kottke TE, Brekke ML, Solberg LI, HughesJR. A randomized trial to increase smoking inter-vention by physicians. JAMA. 1989;261:2101-2106.30. Linn BS. Continuing medical education impacton emergency room burn care. JAMA. 1980;244:565-570.31. Lomas J, Enkin M, Anderson GM, HannahWJ,Vayda E, Singer J. Opinion leaders vs audit andfeedback to implement practice guidelines. JAMA.1991;265:2202-2207.32. Magruder-Habib K, Zung WK, Feussner JR.Improving physicians' recognition and treatmentof depression in general medical care. Med Care.1990;28:239-250.33. Maiman LA, Becker MH, Liptak GS, NazarianLF, Rounds KA. Improving pediatricians' compli-ance-enhancing practices: a randomized trial. AJDC.1988;142:773-779.34. Martin AR, WolfMA, Thibodeaul LA, Dzau V,Braunwald E. A trial of two strategies to modifythe test-ordering behavior of medical residents. NEngl J Med. 1980;303:1330-1336.35. Marton KI, Tul V, Sox HC. Modifying test\x=req-\ordering behavior in the outpatient medical clinic.Arch Intern Med. 1985;145:816-821.36. Mazzuca SA, Vinicor F, Einterz RM, TierneyWM, Norton JA, Kalasinski LA. Effects of the clin-ical environment on physicians' response to post-graduate medical education. Am Educ Res. 1990;27:473-488.37. McConnell TS, Cushing AH, Bankhurst AD,Healy JL, Mcllvenna PA, Skipper BJ. Physicianbehavior modification using claims data: tetracy-cline for upper respiratory tract infection. West JMed. 1982;137:448-450.38. McDonald CJ, Hui SL, Smith DM, et al. Re-minders to physicians from an introspective com-puter medical record: a two-year randomized trial.Ann Intern Med. 1984;100:130-138.39. McPhee SJ, Bird JA, Fordham P, Rodnick JE,Osborn EH. Promoting cancer prevention activi-ties by primary care physicians\p=m-\resultsof a ran-domized, controlled trial. JAMA. 1991;266:538-544.


40. Meyer TJ, Van Kooten D, Marsh S, ProchazkaAV. Reduction of polypharmacy by feedback toclinicians. J Gen Intern Med. 1991;6:133-136.41. Palmer RH, Louis TA, Hsu L-N, et al. A ran-domized controlled trial ofquality assurance in six-teen ambulatory care practices. Med Care. 1985;23:751-770.42. Perera DR, LoGerfo JP, Shulenberger E,Ylvisaker JT, Kirz HL. Teaching sigmoidoscopy toprimary care physicians: a controlled study of con-tinuing medical education. J Fam Pract. 1983;16:785-788.43. Putnam RW, Curry L. Impact of patient careappraisal on physician behaviour in the office set-ting. Can Med Assoc J. 1985;132:1025-1029.44. Putnam RW, Curry L. Physicians' participa-tion in establishing criteria for hypertension man-agement in the office: will patient outcomes be im-proved? Can Med Assoc J. 1989;140:806-809.45. Restuccia JD. The effect of concurrent feed-back in reducing inappropriate hospital utilization.Med Care. 1982;20:46-62.46. Rogers JL, Haring OM, Wortman PM, WatsonRA, Goetz JP. Medical information systems: as-sessing impact in areas of hypertension, obesityand renal disease. Med Care. 1982;20:63-74.47. Rubenstein LV, Calkins DR, Young RT, et al.Improving patient function: a randomized trial offunctional disability screening. Ann Intern Med.1989;111:836-842.48. Sanazaro PJ, Worth RM. Concurrent qualityassurance in hospital care. N Engl J Med. 1978;298:1171-1177.49. Sibley JC, Sackett DL, Neufeld V, Gerrard B,Rudnick KV, Fraser W. A randomized controlledtrial of continuing medical education. N Engl JMed. 1982;306:511-515.50. Stross JK, Bole GG. Evaluation of a continuingeducation program in rheumatoid arthritis. Arthri-tis Rheum. 1980;23:846-849.51. Stross JK, Hiss RG, Watts CM, Davis WK,MacDonald R. Continuing education in pulmonarydisease forprimary care physicians.AmRevRespirDisease. 1983;127:739-746.52. Stross JK, Bole GG. Evaluation of an educa-tional program for primary care practitioners onthemanagement ofosteo-arthritis. Arthritis Rheum.1985;28:108-111.53. Tierney WM, Hui SL, McDonald CJ. Delayedfeedback of physician performance versus imme-diate reminders to perform preventive care. Med

Care. 1986;24:659-666.54. Tierney WM, McDonald CJ, Hui SL, MartinDK. Computer predictions of abnormal test resultseffects on outpatient testing. JAMA. 1988;259:1194\x=req-\1198.55. Tierney WM, Miller ME, McDonald CJ. Theeffect on test ordering of informing physicians ofthe charges for outpatient diagnostic tests. NEnglJ Med. 1990;322:1499-1504.56. Vinicor P, Cohen SJ, Mazzuca SA, et al. DIA-BEDS: a randomized trial of the effects ofphysicianand/or patient education on diabetes patient out-comes. J Chronic Dis. 1987;40:345-356.57. White CW, Albanese MA, Brown DD, CaplanRM. The effectiveness of continuing medical edu-cation in changing the behavior ofphysicians caringfor patients with acute myocardial infarction. AnnIntern Med. 1985;102:689-692.58. White PT, Pharoah CA, Anderson HR, Freel-ing P. Randomized controlled trial of small groupeducation on the outcome of chronic asthma in gen-eral practice. J R Coll Gen Pract Occas Pap. 1989;39:182-186.59. Wilson DM, Taylor W, Gilbert JR, et al. A ran-domized trial of a family physician intervention forsmoking cessation. JAMA. 1988;260:1570-1574.60. Winickoff RN, Coltin KL, Morgan MM, Bux-baum RC, Barnett GO. Improving physician per-formance through peer comparison feedback. MedCare. 1984;22:527-534.61. Wones RG. Failure of low-cost audits with feed-back to reduce laboratory test utilization. Med Care.1987;25:78-82.62. Oxman AD, Guyatt GH. Guidelines for readingliterature reviews. Can Med Assoc J. 1988;138:697\x=req-\703.63. Williams DD, ed. Naturalistic evaluation. In:NewDirectionsforProgramEvaluation. San Fran-cisco, Calif: Jossey-Bass Publishers; 1986;30.64. Fox RD, Mazmanian PE, Putnam RW, eds.Change and Learning in the Lives of Physicians.New York, NY: Praeger Publishers; 1989.65. Oxman AD, Guyatt GH. A consumer's guide tosubgroup analysis. Ann Intern Med. 1992;116:78\x=req-\84.66. Jaeschke R, Singer J, Guyatt GH. Ascertainingthe minimal clinically important difference. Con-trolled Clin Trials. 1989;10:407-415.67. Freiman JA, Chalmers TC, Smith H, KueblerRR. The importance of beta, the type II error andsample size in the design and interpretation of the

randomized controlled trial. N Engl J Med. 1978;299:690-694.68. Fishbein M, Azjen I. Belief,Attitude, Intentionand Behavior: An Introducton to Theory and Re-search. Reading, Mass: Addison-Wesley Publish-ing Co; 1968.69. Elstein AS, Sprafka SA. Medical Problem Solv-ing:AnAnalysis ofClinicalReasoning. Cambridge,Mass: Harvard University Press; 1978.70. Frechette AL, Russo PK. The physician's rolein health promotion\p=m-\asurvey ofprimary-care prac-titioners. N Engl J Med. 1983;308:97-100.71. Soumerai SB, Avorn J. Economic and policyanalysis of university-based drug detailing. MedCare. 1986;24:313-331.72. CarterWB, Belcher DW, Innui TS. Implement-ing preventive care in clinical practice: problemsfor managers, clinicians and patients. Med CareRev. 1981;38:195.73. Knowles MS. The Modern Practice of AdultEducation. Chicago, Ill: AP Follett; 1980.74. EisenbergJM. Sociologic influences on decision\x=req-\making by clinicians. Ann Intern Med. 1979;90:957.75. Tugwell P, Dok C. Medical record review. In:Neufeld VR, Norman GR, eds. Assessing ClinicalCompetence. New York, NY: Norman G. Springer;1985:142-182.76. Davis DA, Norman GR, Premi J, et al. At-tempting to ensure physician competence. JAMA.1990;263:2041-2042.77. Parboosingh J, Avard D, Lockyer J,Watson M,Pim C, Yee J. The use of clinical recall interviewsas a method of determining needs in continuingmedical education. Proc Conference Res Med Educ.1987;26:103-108.78. Hiss RG, MacDonald R, Davis WK. Identifica-tion of physician educational influentials in smallcommunity hospitals. Proc Conference Res MedEduc. 1978;17:283-288.79. Stinson ER, Mueller DA. Survey of health pro-fessionals information habits and needs. JAMA.1980;243:140-143.80. Curry L, Putnam RW. Continuing medical ed-ucation in Maritime Canada: the methods physi-cians use, would prefer and findmost effective. CanMed Assoc J. 1981;124:563-566.81. Richards RK. Physicians' self-directed learn-ing: a new perspective for continuing medical ed-ucation, I: reading. Mobius. 1986;6:1-13.

The first edition of the Continuing Medical Education Directory, published in August 1992, consolidates a va¬riety of information needed for attending, planning, accrediting, and delivering continuing medical education op¬portunities for physicians. In addition to lists of accredited sponsors in the United States and Canada and of con¬tinuing medical education opportunities offered by those sponsors, the Directory provides a Federation meetingcalendar, with current addresses and phone numbers. The Directory also documents continuing medical educa¬tion reporting requirements of state licensing boards, national and state medical societies, and specialty boards.Recent developments in comercially-supported continuing medical education are summarized, and the guidelinesof several national organizations—the American Medical Association, the Accreditation Council for ContinuingMedical Education, the American College of Physicians, and the Society of Medical College Directors of Continu¬ing Medical Education—are reprinted. The Directory summarizes the structure and functions of the AccreditationCouncil for Continuing Medical Education and reproduces the Essentials and Guidelines for Accreditation ofSponsors of CME and Procedures for ACCME Accreditation. Additional features include descriptions of the AMAPhysician's Recognition Award Program (along with the application itself), the research database maintained byMcMaster University in Hamilton, Ontario, and special continuing medical education activities. (Order from 800621-8335; $36.00 for AMA members, $45.00 nonmembers. For other information, call Enza Messineo, 312 464-4952.)


evidence for the effectivenessof - podiatry for the...enee, or inconclusive. statisticallysig¬...

Documents