Does it Matter —
Timeliness or Accuracy of Results?
Results of a Research Program on Rapid Reviews
Andrea C. Tricco PhD MSc
CADTH Symposium 2015
I have no actual or potential conflict of
interest in relation to this presentation.
Conflict of interest
2
Background
Scoping Review of Rapid Reviews
International Survey of Rapid Review Producers
International Consensus-building Exercise with Key
Stakeholders
Ultimate Goal of Our Research Program
Questions
Outline
2
Background
Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [Khangura 2012]
Evidence suggests that decision-makers are currently using rapidreviews to inform their decision-making processes
Few studies have examined the methodological characteristics of rapid reviews
We aimed to conduct a research program on rapid reviews to clarify the methods and perceptions of rapid review approaches
3
Objective & methods
Objective – To examine rapid review approaches, guidance, impact, and comparisons
through a scoping review
Methods– Used methodologically rigorous scoping review methods proposed by
Arksey and O’Malley (2005)
– MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review
producers, and reference lists were searched to identify articles for inclusion
– Two reviewers independently screened citations and full-text articles, and
abstracted data
5
Results
Study flow figure
N=3397 citations from MEDLINE, EMBASE, Cochrane Library,
previous systematic reviews, and grey literature
N=3135 excluded titles and abstracts
N=100 rapid reviews(plus 1 companion report)
N=262 potentially relevant full-text articles
N=161 excluded full-text
reports
6
Study CharacteristicsNo. of Rapid Reviews
(n=100)
Year of Publication
1997-2000 2
2001-2004 10
2005-2008 30
2009-2012 51
2013 5
Not Reported 4
Corresponding
author’s
continent
Europe (including UK) 58
North America (Canada & United States) 20
Australia 15
Multiple continents 3
Asia 1
South America 1
Not Reported 2
Study characteristics
Results (continued)
8
Results (continued)Study Characteristics No. of Rapid Reviews (n=100)
Article Type
Application (82 with methods) 84
Development 7
Impact 6
Comparison 4
Topic of Review
Intervention 62 (74%)
Frequency 10 (12%)
Causal association 4 (5%)
Diagnosis 4 (5%)
Patient experience 2 (2%)
Screening 2 (2%)
Not applicable 16
Study characteristics (continued)9
Results (continued)
Methods characteristics
4%
2%
83%
70%
51%
22%
12%
17%
35%
25%
10%
17%
23%
98%
12%
24%
10%
11%
68%
49%
54%
46%
68%
57%
73%
6%
6%
40%
67%
20%
34%
17%
29%
22%
26%
0% 20% 40% 60% 80% 100%
Duration of review
Protocol mentioned
Databases Searched
Grey literature search
Scanning references
Contacting authors
Date limit
Language limit
Titles and abstracts screening
Full-texts screening
Data abstraction
Quality appraisal
SR method Streamlined method Not reported
10
Results (continued)
Approach Literature search Search limit Screening Data abstractionRisk of bias
appraisal
1>1 database,
published only
Both date and
languageOne reviewer
One person
abstracts, other
verifies
One person
assesses, other
verifies
2
Updating the literature
search of a previous
review, published only
None One reviewer One reviewer Not performed
3>1 database, grey
literature
Both date and
languageOne reviewer One reviewer Not performed
4>1 database, grey
literature
Either date or
languageOne reviewer One reviewer Not performed
5>1 database, grey
literatureDate One reviewer One reviewer One reviewer
5 most frequent rapid review approaches11
Conclusions
Numerous rapid review approaches were identified
Little consistency exists in the field
Poor quality of reporting was observed
Prospective study comparing the results from rapid reviews to
those obtained through systematic reviews is warranted.
12
Objective and methods
Objective– To determine different rapid review approaches used by rapid review
producers
Methods– International survey of 63 organizations administered via FluidSurvey
– Survey pilot-tested prior to administration
– Reminders to non-respondents sent every 2 weeks
– Contacted through postal mail if no response
– $10 financial incentive from Amazon
14
Results
Study flow figure of participants
63 organizations contacted
41 responses (65%)
22 did not respond
15
Results (continued)
Summary of rapid review characteristics
Review Characteristics Count (%)
Duration of Review
(weeks)
1-12 62 (70)
12-26 18 (20)
26-36 6 (7)
≥ 52 2 (2)
Commissioning Agency
Government Agencies & Health Ministries 69 (78)
Healthcare Organizations, Hospitals & Community Health 51 (58)
Healthcare Professionals 13 (15)
Industry 4 (5)
Target Audience
Government Agencies & Health Ministries 73 (83)
Healthcare Professionals 46 (52)
Patients 19 (22)
Researchers 21 (24)
16
Results (continued)
Rationale provided Count (%)
Decision-maker timeline 57 (66)
Focused or brief question 8 (9)
Lack of resources 5 (6)
Increase efficiency (including timeliness) 4 (5)
Broad understanding of an area 4 (5)
Identify topics requiring a systematic review 2 (2)
Update a systematic review 2 (2)
Well-established intervention 1 (1)
Evidence is unclear 1 (1)
Rationale for conducting rapid reviews17
Results (continued)Review Stage Most frequent streamlined approach Count (%)
Identifying relevant studies Used previous review(s) as a starting point 79 (92)
Limitations on search strategy Limited review by date of publication 75 (88)
Study selection Screening conducted by ONE reviewer only 68 (85)
Data Abstraction Data abstraction performed by ONE reviewer only 67 (84)
Quality (risk of bias) appraisal process
Risk of bias assessed by ONE reviewer only 68 (86)
Synthesis Narrative summary 75 (90)
Summary results of most frequently streamlined approaches 18
Conclusions
Results are consistent with scoping review of rapid reviews
Rapid reviews usually conducted in 1-12 weeks
Government agencies and health ministries are primary
commissioners
Many different streamlined methods are being used.
19
International consensus-building
exercise regarding rapid reviewsSubmitted to the J of Clin Epi
Methods project 3:
Objective & methods
Objective – To conduct a consensus-building exercise to select a rapid review approach
that will be prospectively tested in a diagnostic study
Methods– Editors, healthcare providers, researchers, policy-makers, and industry
stakeholders (including participants of the CADTH rapid review summit)
– Asked to rank the 5 most frequent rapid review approaches identified in our
scoping review and survey using FluidSurvey
– Results presented to participants, followed by a facilitated discussion (online
and in-person) and re-ranking exercise using FluidSurvey
21
Results
Study flow figure of participants
26 individuals
contacted
113 responses (72%)
3 did not respond
130 individuals
contacted
40 did not respond
22
Online delphi In-person delphi
Summary of ranking results by approach
Results (2)Rapid review approach
Feasibility Timeliness Comprehensiveness Risk of Bias
Approach 1 1st 2nd 4th 1st
Approach 2 2nd 1st 5th 5th
Approach 3 3rd 3rd 3rd 2nd
Approach 4 4th 4th 2nd 4th
Approach 5 5th 5th 1st 3rd
*Ranked based on the distribution of "very" and "extremely" on the 7-point Likert scale, except Risk of Bias was ranked on distribution of “not at all” and “very”
23
The highest ranked method was: Approach 1
‒ Most feasible (72%, n=81 out of 113 responses)
‒ Lowest perceived risk of bias (12%, n=12 out of 103)
‒ 2nd in timeliness (37%, n=38 out of 102)
‒ 5th in comprehensiveness (5%, n=5 out of 100)
We will use the information from the consensus-building exercise
to select the rapid review approach for a prospective study.
Conclusion
24
Ultimate goal of this research
“We can give you results within 4 months, but the meta-
analysis estimates will be inaccurate by 35%”
Rapid review definition (Shannon Kelly)
Identify 5 frequently used methods
Diagnostic study to test a rapid review approach
Identify and characterize rapid review methods
25
Will use these results from our research program to inform a
diagnostic study:
− Index test: Rapid Review Approach
− Reference standard: Systematic Review
Collaboration between 3 Canadian Knowledge Synthesis Centers
Targeting CIHR and PCORI
Proposed diagnostic study
Diagnostic Accuracy of Rapid reviews compared To
Systematic reviews (DARTS)
26
Summary
This research program provides up-to-date information on rapid
review methods reported in the literature, as well as stakeholder
experiences and perceptions regarding rapid reviews
Poor quality of reporting was observed in the literature
Rapid reviews have many names and approaches and some
methods might be more desirable than others
A prospective study comparing the results of rapid reviews to those
obtained through systematic reviews is necessary.
27
Acknowledgements
Funding: Canadian Institutes of Health Research/Drug Safety
and Effectiveness Network
− Operating grant to update 2 systematic reviews,
international survey, Delphi
− New investigator award
Research team: Jesmin Antony, Wasifa Zarin
Co-investigators: Drs. Straus, Moher, Hutton, Sherifali
28