how do cme speakers use research results to support
TRANSCRIPT
How do CME speakers use research results to support therapeutic
recommendations?A quantitative and qualitative study
Michael Allen, Tanya Hill, Richard Handfield-Jones, Mike Fleming, Doug Sinclair, Tom Elmslie
SACMERancho Mirage, CA
April, 2009
3
Research TeamI’m Mike Allen,
the PI from Dalhousie CME
I’m Richard Handfield-Jones,
the PI from University of Ottawa CME. I’m in the army now and AWOL.
I’m Mike Fleming from Dalhousie CME and I’m
not sure how I got involved in this.
I’m Doug Sinclair from Dalhousie CME. I can
find time to do anything.I’m Tanya Hill the research associate from Dalhousie CME. I don’t care who says they’re the PI, I’m running this
show.
I’m Tom Elmslie. I’m an ideas man and helped with
experimental design.
4
Funding
Royal College of Physicians and Surgeonsof Canada
Nova Scotia Department of Health throughDrug Evaluation Alliance of Nova Scotia
6
Think about this RCT …
Outcomes: non-fatal myocardial infarction and death from coronary heart disease Subjects: No history of coronary heart disease (primary prevention)Duration: 3.3 years N: 5100 patients in the control and 5100 patients in the drug groupThe patient characteristics are:
80% male mean age = 63 yrs. mean blood pressure = 164/95
ASCOT Lancet 2003;361:1149-58
7
1. The drug led to a 36% decrease in the incidence of non-fatal MI and CHD death (relative risk reduction).
2. The drug decreased the rate of non-fatal MI and CHD death from 3.0% to 1.9%, an absolute risk reduction of 1.1%.
3. You would have to treat 94 patients for 3.3 years to prevent one non-fatal MI or a death from CHD (number needed to treat).
4. The 95% confidence intervals around the previous result (ie, treat 94 patients for 3.3 years to avoid one non-fatal MI or CHD death) are 60 and 215.
5. At the end of 3.3 years, 97.0% of patients who don't take the drug will remain free of a cardiac event and 98.1% of patients who take the drug will remain free of a cardiac event (inverse absolute RR).
What is your interpretation of the following results?How likely might you be to prescribe or take the drug based on each one?
8
Patients not having MI or dying: Placebo
Patients having MI or CHD death
Patients NOT having
MI or CHD death
9
Patients not having MI or dying: Treatment
Patients having MI or CHD death
Patients NOT having
MI or CHD death
10
Forrow L, Taylor WC, Arnold RM. Absolutely relative: How research results are summarized can affect treatment decisions. Am J Med 1992; 92:121-4.
Naylor CD, Chen E, Strauss B. Measured enthusiasm: Does the method of reporting trial results alter perceptions of therapeutic effectiveness? Ann Intern Med 1992; 117:916-21.
Nikolajevic-Sarunac J, et al. Effects of information framing on the intentions of family physicians to prescribe long-term hormone replacement therapy. J Gen Intern Med 1999; 14(10):591-598
Nexoe J, Kristiansen IS, Gyrd-Hansen D, Nielsen JB. Influence of number needed to treat, costs and outcome on preferences for a preventive drug. Fam Pract 2005; 22(1):126-131.
11
ObjectivesTo determine:
How much emphasis speakers place on presenting data in absolute vs relative termsHow familiar FPs and speakers are with statistical terms (especially absolute and relative terms)The attitudes and preferences of FPs about having research data presentedHow well speakers link therapeutic recommendations to research data
12
Methods
2 annual 3-day Family Medicine review conferencesHalifax (Dalhousie University CME)- 230 registrants- 42 presentations (lecture and concurrent sessions)
Ottawa (University of Ottawa CME)- 209 registrants- 38 presentations (lecture and concurrent sessions)
3 data collection methods for each conference
13
Methods – Data Collection1. Questionnaire
Registrants and SpeakersEvaluate understanding of statistical terms
2. Focus groups1.5 hour with FPs from each siteDetermine preferences about types of statistics used
3. Analysis of presentationsPowerPoint filesVideo recording of presentations
14
Methods – Data Analysis1. Questionnaire
Descriptive statistics
2. Focus groupsRecorded and transcribedContent analysis
3. Analysis of presentationsPowerPoint files - How research data presented – slide counts
Videorecording- Linking of research data to recommendations – Likert scales
Tworesearchers
15
Results: Questionnaire
Registrants (N=121; 29% response rate)97% physicians; 81% certificant of CFPC51% male; 49% femaleAvg. yrs practice = 18.027% completed EBM workshop in last 10 years
Speakers (N=20; 33% response rate)19 specialists, 1 family physician70% male; 30% femaleAvg. yrs practice = 18.030% completed EBM workshop in last 10 yrs
17
Results: Questionnaire (Registrants N=121)
% understand vs. % correct response
5136 40
4755
38
0
20
40
60
80
100
NNT ARR RRR
% understand andcould explain
% correctlycalculated
18
Results: Questionnaire (Speakers N=20)
% understand vs. % correct response
80
45 50
75 70 65
0
20
40
60
80
100
NNT ARR RRR
% understandand couldexplain
% correctlycalculated
19
Results: Questionnaire
06
3438
22
05
40
3025
0
10
20
30
40
50
60
Very little Some A lot
Registrant(N=121)
Speaker(N=20)
"How much emphasis should CME speakers place on presenting research results when making therapeutic recommendations?"
Per
cent
resp
onse
s
20
Results: Focus group
2 focus groups9 males; 5 females
Most preferred statistics:1. Numbers needed to treat 2. Inverse ARR3. Confidence Intervals
22
Results: Focus group
Trust in speakers and CME providersTrust that specialists can critically evaluate researchSkepticism about specialist ability to interpret dataAttend CME because specialists simplify data and make recommendations supported by research Trust CME offices to ensure programs present data accurately and completelyDisclose conflict of interest
24
Preliminary results: PowerPoints (N=26)
General termsGraphsRisk (event rate)PrevalenceAbsolute or relative not specified
Absolute termsAbsolute risk reduction or increaseNumber needed to treat or harm
Relative termsRelative riskRelative RROdds ratioHazard ratio
95% confidence intervals
25
Preliminary results: PowerPoints (N=26)
PresentationTotal
content slides
Therapeutic recmdation
General terms
Relative terms
Absolute terms 95% CIs
1 29 4 9 0 0 02 49 14 0 0 0 03 45 10 15 0 0 04 72 22 10 1 0 25 37 5 1 0 0 06 16 3 0 0 0 07 111 14 27 4 0 28 131 23 18 10 6 9
Number of slides with . . .
26
Preliminary results: Powerpoints (N=26)
Presentation Total Therapeutic recmdation
General terms
Relative terms
Absolute terms 95% CIs
9 49 6 3 4 2 110 26 7 6 3 0 111 32 7 10 3 1 212 21 7 4 0 0 013 49 16 2 0 0 014 69 22 24 5 1 115 39 2 10 2 4 216 102 8 4 0 0 017 51 9 10 0 0 018 48 12 0 0 0 0
Number of slides with . . .
27
Preliminary results: PowerPoints (N=26)
Presentation Total Therapeutic Recmdation
General terms
Relative terms
Absolute terms 95% CIs
19 31 10 5 0 0 0
20 50 7 2 0 0 0
21 29 6 0 0 0 0
22 68 8 8 5 4 4
23 21 0 1 0 0 0
24 51 3 3 0 0 0
25 59 3 6 5 0 3
26 59 22 12 5 0 2
TOTAL 1344 250 200 47 18 29
Number of slides with . . .
28
Preliminary results: Presentations (N=26)
PresentationTotal
content slides
Therapeutic recmdation
General terms
Relative terms
Absolute terms 95% CIs
1 29 4 9 0 0 02 49 14 0 0 0 03 45 10 15 0 0 04 72 22 10 1 0 25 37 5 1 0 0 06 16 3 0 0 0 07 111 14 27 4 0 28 131 23 18 10 6 9
Number of slides with . . .
29
Preliminary results: Videotapes (n=19)
Scale: 1=very little, 3=some, 5=a lot *n/a = 13 presentations
Mean* (SD)
1. How much time did speaker spend on therapeutic interventions of any kind? 4.0 (.82)
2. How much time did the speaker spend relatively between relative/absolute terms?* 3.2* (.41)
3. How much time/effort did the speaker take to explain the research data? 2.8 (1.4)
4. How often did the speaker mention relative/absolute terms when not on slide? 1.7 (1.2)
5. How thoroughly did the speaker link relative/absolute terms with recommendations? 2.1 (1.2)
6. How much evidence did the presentation warrant? 3.4 (.90)
7. How much emphasis did the CME speaker place on presenting research results when making therapeutic recommendations? 2.9 (1.4)
30
ConclusionsMost FPs have low to moderate level of statistical comprehension
Speakers have higher level of statistical comprehension than FPs
FPs want recommendations to be supported with research
FPs want speakers to critically evaluate the research
Most speakers present little data in relative or absolute terms or with 95% CIs
Focus group recommendations for CME providersProvide short review on statistics for FPsDevelop consistent format for presentation of research data
31
Next Steps
Provide CME for FPs to help them understand research dataDevelop format for presenting data that FPs find helpful and understandableProvide faculty development for speakersAdd question to evaluation form re presentation of research data
32
Questions?