joanne yaffe, phd, university of utah paul montgomery, phd, university of oxford
DESCRIPTION
How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in the development of reporting guidelines. Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford - PowerPoint PPT PresentationTRANSCRIPT
How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in
the development of reporting guidelines
Joanne Yaffe, PhD, University of UtahPaul Montgomery, PhD, University of Oxford
Sally Hopewell, PhD, UK Cochrane CentreLindsay Shepard, MSc, University of Utah
The Empty Reviews Project is funded by the Cochrane Opportunities Fund
Summary of Presentation
• The Problem of Empty Reviews• Current State of Cochrane Empty
Reviews and Apparent Direction of Travel• How this Project Aims to Proceed• Cochrane Contributor Inputs
The Cochrane Empty Reviews [email protected]
Introduction: Why study empty reviews?
• First, what do we mean by Empty Review?• An empty review is a systematic
review that reports no studies eligible for inclusion.
The Cochrane Empty Reviews [email protected]
• Lack of high quality studies to address question posed. –Very narrow population–Very focused intervention–Outcomes that are not measured.
• Many plan empty reviews to justify new trials in response to funding policies.
The Cochrane Empty Reviews [email protected]
How do systematic reviews come to be empty?
• Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions.
• Should observations be made based on excluded studies to provide decision-makers with some guidance? (following Lang, Edwards & Flieszer 2007)
The Cochrane Empty Reviews [email protected]
• The Cochrane Handbook (2009) does not address the reporting of empty reviews specifically, although it does address excluded studies in general.
The Cochrane Empty Reviews [email protected]
“A Cochrane review includes a list of excluded studies, detailing any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may on the surface appear to meet the eligibility criteria but on further inspection do not, and also those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible…”
The Cochrane Empty Reviews [email protected]
Excluded Study Guidance in Cochrane Handbook (p. 154)
• Concern: Empty reviews may leave decision-makers dependent on weaker research which may be presented without sufficient discussion of limitations.
• Concern: Empty reviews may fail to provide any implications either for practice or research.
The Cochrane Empty Reviews [email protected]
• Empty Reviews Project Aim: To develop guidelines, in consultation with stakeholders and the editors of the Cochrane Handbook, for authors of reviews who do not find studies eligible for inclusion.
The Cochrane Empty Reviews [email protected]
Prevalence of Empty Reviews in The CDSR (as of 15 August 2010)
• Total reviews in The CDSR
N= 4320• Total empty reviews in The CDSR
N= 375 (8.7%)
The Cochrane Empty Reviews [email protected]
Number of Empty Reviews across CRGs
The Cochrane Empty Reviews [email protected]
0
5
10
15
20
25
30
35
40
Proportion of Empty Reviews across CRGs
The Cochrane Empty Reviews [email protected]
0.0
5.0
10.0
15.0
20.0
25.0
30.0
Empty Reviews by Year of First Publication
The Cochrane Empty Reviews [email protected]
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100
10
20
30
40
50
60
70
Age of Empty Reviews in The CDSR
• Half of all empty reviews are 3 years or older, with some dating to 1995.• Mean Age = 3.3 years (SD 3.4)
The Cochrane Empty Reviews [email protected]
Time since last update of Empty Reviews
• 48.3% of all empty reviews have been updated within the last 2 years, but 5.9% (21) have not received an update in 7 years or longer.• Mean Time since Update = 2.8 years
(SD 2.2)
The Cochrane Empty Reviews [email protected]
How many excluded studies are listed in these empty reviews?
• 95 (25.3%) of empty reviews do not list excluded studies.• The number of excluded studies
ranges from 0 to 124. • Mean number of excluded studies =
9.6 (SD 14.5)The Cochrane Empty Reviews [email protected]
How many studies are listed as pending assessment?
• 360 (96.0%) of empty reviews do not list pending studies.• The number of pending studies
ranges from 0 to 9. • Mean number of pending studies =
0.1 (SD 0.7)The Cochrane Empty Reviews [email protected]
How many studies are listed as on-going?
• 338 (90.1%) of empty reviews do not list on-going studies.• The number of on-going studies
ranges from 0 to 4. • Mean number of on-going studies = 0.1
(SD 0.5)
The Cochrane Empty Reviews [email protected]
How many reviews list no studies at all?
• 88 (23.5%) of empty reviews do not list any excluded, pending, or on-going studies.
The Cochrane Empty Reviews [email protected]
So what?• So, what are the implications of all these empty
reviews? The lack of guidelines for reporting studies that have no
included studies may result in inconsistently reported reviews.
Policy-makers and practitioners might not rely on Cochrane reviews for decision-making.
More importantly, however, the lack of guidelines for authors of empty reviews may result in less than useful conclusions and further, may misstate results.
The Cochrane Empty Reviews [email protected]
What do we mean by “less than useful conclusions”?
The Cochrane Empty Reviews [email protected]
Less than useful conclusions.
What sorts of conclusions might be less than useful?
• Reference to single studies without appropriate caveats.• Unsupported statements.• No statements at all.
The Cochrane Empty Reviews [email protected]
Review A : Implications for practiceSufficient evidence based on randomised controlled trials is lacking and therefore we can draw no reliable conclusions about the comparative effectiveness of medical and surgical interventions of [deleted to de-identify]. However, the studies identified, but excluded from this review, suggest there may be a considerable response to placebo treatment in this condition.
Review B: Implications for practiceOn the evidence currently available, [ ] cannot be recommended for the treatment of [ ].
Implications for research[ ] has shown promise in animal studies but there are no randomized double-blind placebo-controlled trials in people with [ ]. Such trials are needed.
The Cochrane Empty Reviews [email protected]
What might be better ways to handle conclusions?
• Specific recommendations for design of trials.
• Appeal to related reviews, drawing analogy to present question, with caveats.
• Balanced discussion of weaker studies.
The Cochrane Empty Reviews [email protected]
Review C: Implications for research 1. General
As with all similar reviews, public registration of a study before anyone is randomised would ensure that participants could be confident that people would know that the study had at least taken place. Compliance with CONSORT (Moher 2001), both on the part of authors and editors, would help to clarify methodology and many outcomes. Failure to comply with CONSORT guidelines results in loss of data and confusion in results, neither of which helps clinicians, patients or managers.
2. 2. Specific [ ] is a most distressing [ ] disorder that remains highly prevalent, both in the developed and developing world. This review highlights the need for well designed, conducted and reported clinical trials (Table 1) to address the claims of open studies as regards the effects of the [ ] group of drugs for [ ].
The Cochrane Empty Reviews [email protected]
What do we hope to accomplish?
• Consistency in the way that excluded studies are addressed in the authors’ conclusions.
• Collaboration with other stakeholders towards development of guidelines for inclusion in The Cochrane Handbook.
The Cochrane Empty Reviews [email protected]
Our Plan1. Complete our analysis of the 375 empty
reviews in The CDSR as of 15 August, 2010.
2. Solicit input from all interested parties via [email protected] and www.empty-reviews.org (coming soon!).
3. Develop specific recommendations around the writing, presentation, and persistence in updating empty reviews.
The Cochrane Empty Reviews [email protected]
4. Present our findings and recommendations at a consensus meeting of key stakeholders in Spring, 2011, including Cochrane Handbook editors.
5. Use the results of the consensus meeting to refine our recommendations.
6. Present our findings to the Madrid Colloquium in October 2011, for further discussion.
The Cochrane Empty Reviews [email protected]
The Cochrane Empty Reviews Project Funded by The Cochrane
Opportunities Fund
Paul Montgomery, PhD, University of Oxford, PISally Hopewell, PhD, UK Cochrane Centre, Co-PIJoanne Yaffe, PhD, University of Utah, Co-PILindsay Shepard, MSc, University of Utah, Research Associate
[email protected]://www.empty-reviews.org