assessing the impact of searching fewer databases in rapid · of searching fewer databases in rapid...

Trusted evidence.

Informed decisions.

Better health.

Irma Klerings

Cochrane Austria

[email protected]

Assessing the impact

of searching fewer

databases in rapid

reviews

2

Potential conflicts of interest

• This research was funded by Cochrane Austria.

• No other potential or actual conflicts of interest in

relation to this presentation.

Decision making needs

reliable evidence syntheses

4

Systematic Reviews

• Most reliable &

valid support for

decision-making

• Synthesis of all

evidence about a

research question

• Systematic methods

minimize bias

• Takes 6-12 months

to complete

Rapid Reviews

• Based on systematic

review methods:

processes are

accelerated and

methods are

streamlined

• Takes 5–12 weeks to

complete

• Reliability of

conclusions?

5

Do bodies of evidence that are

based on abbreviated literature searches

lead to different conclusions

compared with those based on

comprehensive, systematic literature

searches?

Research question

6 METHODS

7

Sample selection

60 randomly selected Cochrane

reviews

Main inclusion criteria

• Authors were able to draw a conclusion

• Summary of findings tables

• Reproducible Meta-analyses

• Used MEDLINE, EMBASE, and Central

8

14 search abbreviated search approaches

compared to original comprehensive search

Search of reference lists of relevant publications

MEDLINE EMBASE CentralMEDLINE + EMBASE

MEDLINE + Central

Central + EMBASE

MEDLINE + Central + EMBASE

+

Database

coverage# of included references indexed in a database

# of included references cited in the reviewx100

Recall# of included references retrieved by a search approach

# of included references cited in the reviewx100

9

Research Process

Run searches

Check partiallyfoundstudies

RecalculateMA & createnew SoF-tables

Web-basedsurveyCochraneauthors

Non-inferiorityanalysis

10

Non-inferiority analysis

Scenario 2:

non-inferior

Scenario 1:

inferior

Non-inferiority margin(Wagner et al. 2017)

11 RESULTS

12

Review characteristics

Type of intervention

• Pharmacological (drugs, vaccines) 30 (50%)

• Non-pharmacological (psychological, educational,

dietary, physical exercise, complex interventions,

screening, surgery, management strategies) 30 (50%)

Information sources

• Medline, Embase, Central 60 (100%)

• At least 2 other types of information sources 60 (100%)

Other bibliographic databases 56 (93%)

Grey literature and unpublished data 59 (98%)

Other sources (reference lists, citation tracking,

handsearch)

56 (93%)

Study design of included studies

• RCT only or RCT+quasi-RCT 53 (88%)

• RCT+controlled clinical trial, before-after study or

interrupted time series 7 (12%)

13

Database coverage

Were the included references indexed in any of the databases?

72%

63%

75% 75%

89% 89%

92%93% 92%

100% 100% 100% 100%100%

87%

82%

91%94%

96% 97%99%

40%

50%

60%

70%

80%

90%

100%

Medline Embase Central M+E M+C C+E M+E+C

Refe

ren

ce

s ind

exe

d

Median & interquartile range of database coverage (n=60)

MedianM=Medline E=Embase C=Central

14

Recall

Were the included references found by the search strategies?

57%

71%

47%

63%

65%

75%

67%

77%

74%

79%

73%

79%78%

85%

87%

90%

84%

90%

93%

95% 95%97%

94%

98% 98%100% 100% 100%

76%

84%

71%

80%

75%

82%

85%

90%

86%

90% 89%92%

90%

94%

40%

50%

60%

70%

80%

90%

100%

Re

fere

nce

s fo

un

d

Median & interquartile range of search strategy recall without/with addedreference list checking (n=60)

MedianM=Medline E=Embase C=Central R=Reference list checking

15

Impact on overall conclusion if

Discordant conclusion = any change in conclusion

• less certainty, but the same direction of

conclusion

• opposite conclusion (= changed direction of

conclusion)

• no conclusion possible

Conclusions of

abbreviated searches

M M+R E E+R C C+R M+EM+E+

RM+C

M+C+

RC+E

C+E+

R

M+C+

E

M+C+

E+R

conclusion

does not

change48 50 44 49 47 47 50 53 53 53 50 54 52 55

Same

conclusion

with less

certainty

6 4 6 5 8 8 5 2 3 3 6 2 4 1

Opposite

conclusion2 2 3 1 1 1 1 1 1 1 1 1 1 1

conclusion

is no

longer

possible

4 4 7 5 4 4 4 4 3 3 3 3 3 3

M=Medline E=Embase C=Central R=Reference list checking

17

Any change in conclusion

11%

8%

16%

10%

12% 12%

8%

5% 5% 5%

8%

6%

4%3%

32%

29%

40%

30%

34% 34%

29%

23% 23% 23%

29%

25%

21%

18%20%

17%

27%

18%

22% 22%

17%

12% 12% 12%

17%

13%

10%8%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Proportion of conclusions with any changes and 95% confidenceinterval for each search approach (without/with added reference list

checking) (n=60)

Proportion of discordant conclusion

non-inferiority margin


18

Impact on overall conclusion if

Disconcordant conclusion

= Opposite conclusion only

19

Opposite conclusions only

0% 0%

1%

0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

12% 12%

14%

9% 9% 9% 9% 9% 9% 9% 9% 9% 9% 9%

3% 3%

5%

2% 2% 2% 2% 2% 2% 2% 2% 2% 2% 2%

0%

2%

4%

6%

8%

10%

12%

14%

16%

Proportion of opposite conclusions and 95% confidence interval foreach search approach (without/with reference list checking) (n=60)

Proportion of discordant conclusion

non-inferiority margin


20 • Same conclusion,

less certain

• Opposite

conclusion

• Conclusion

unchanged

• No conclusion

possible

Conclusion stays the same

Refe

rences fo

un

d

(n=60)

21

Pharmacological vs non-pharma. reviews

23%

20%

37%

27%

33%

33%

20%

13%

17%

17%

27%

20%

13%

10%

17%

13%

17%

10%

10%

10%

13%

10%

7%

7%

7%

7%

7%

7%

MEDLINE

MEDLINE +Refs

Embase

Embase +Refs

CENTRAL

CENTRAL +Refs

MEDLINE +Embase

MEDLINE +Embase +Refs

MEDLINE +CENTRAL

MEDLINE +CENTRAL +Refs

CENTRAL +Embase

CENTRAL +Embase +Refs

MEDLINE +CENTRAL +Embase

MEDLINE +CENTRAL +Embase+Refs

Proportion of "conclusions with any change" in reviews on non-pharmacological interventions (n= 30)

Proportion of "conclusions with any change" in reviews on pharmacological interventions (n= 30)

22

Recall pharma. vs non-pharma reviews

76%

83%

75%

81% 82%

90%

85%89% 88%

92% 93% 94% 94% 96%

40%

50%

60%

70%

80%

90%

100%

M M+R E E+R C C+R M+E M+E+R M+C M+C+R C+E C+E+R M+C+E M+C+E+R

Median recall of pharmacological reviews (n=30)

76%

84%

61%

72% 71%

78%

85%

92%

80%

88%

81%

85%87%

93%

40%

50%

60%

70%

80%

90%

100%


Median recall of non-pharma. reviews (n=30)


23

Conclusions depending on number of

included studies

36%

27%

41%

32%

32%

32%

27%

18%

18%

18%

27%

23%

14%

14%

11%

11%

18%

11%

16%

16%

11%

8%

8%

8%

11%

8%

8%

5%

MEDLINE

MEDLINE +Refs

Embase

Embase +Refs

CENTRAL

CENTRAL +Refs

MEDLINE +Embase

MEDLINE +Embase +Refs

MEDLINE +CENTRAL

MEDLINE +CENTRAL +Refs

CENTRAL +Embase

CENTRAL +Embase +Refs

MEDLINE +CENTRAL +Embase

MEDLINE +CENTRAL +Embase +Refs

Proportion of "conclusions with any change" in reviews including fewer than ten primary studies (n=22)

Proportion of "conclusions with any change" in reviews including ten or more primary studies (n=38)

24

Recall: number of included studies

67%

75%

63%

70%74%

78%

71%

80%

75%

80%78%

81% 80%83%

40%

50%

60%

70%

80%

90%

100%


Median recall of reviews with fewer than ten included studies (n=22)

81%

87%

75%

83%80%

88% 87%91%

89%92% 93% 94% 94% 96%

40%

50%

60%

70%

80%

90%

100%


Median recall of reviews with ten or more included studies (n=38)


25 DISCUSSION

26

Conclusion

• If decision-makers are willing to accept less certainty

and a small risk for opposite conclusions, some

abbreviated searches are viable options for rapid

evidence syntheses.

• Decisions demanding high certainty require

comprehensive searches.

• Impact of abbreviated searches depends on type of

intervention, „size“ of the topic, and definition of „changed

conclusion“

27

Discussion

Limitations:

Central is only useful for RCTs

External validity (raw database entries vs. real-life)

Reference entries

Points for discussion:

Limiting the number of databases searched could be

more suitable for rapid reviews of pharmacological

interventions

Different streamlined methods for different

intervention-types?

28

More information

Study protocol:

Nussbaumer-Streit, B., I. Klerings, G. Wagner, et al. (2016). "Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study." Syst Rev 5(1): 197.

Main analysis:

Nussbaumer-Streit, B., I. Klerings, G. Wagner, et al. (2018). "Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study." J Clin Epidemiol Article in Press.

29

Principal investigators:

Barbara Nussbaumer-Streit, Gerald Gartlehner

Search specialists:

Irma Klerings, Megan van Noord, Tarquin Mittermayr

Project members:

Andreea Dobrescu, Thomas Heise, Jan Stratil, Gernot Wagner, Susan Armijo Olivo,

Emma Persad, Stefan Lhachimi, Hajo Zeeb, Birgit Teufer

Meta-epidemiology:

Lars Hemkens, Aviv Ladanie

Thank you to

the authors of the Cochrane Reviews &

our team

assessing the impact of searching fewer databases in rapid · of searching fewer databases in rapid...

Documents