Reproducibility of Search Strategies in Systematic ReviewsPreliminary Results

Jonathan Koffel, MSIBio-Medical Library, University of Minnesota

Melissa Rethlefsen, MLIS AHIPEccles Health Sciences Library, University of Utah

Rise of the Systematic Review

2003 2004 2005 2006 2007 2008 2009 2010 2011 20120

1000

2000

3000

4000

5000

6000

7000

8000

9000Meta-Analysis[pt]

DARE Citations

Reporting Guidelines:PRISMAMOOSECochraneInstitute of Medicine

Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.

Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 2009

“The Scopus, EBSCOhost, Ovid, and Web of Science platforms were searched.”

Pediatrics. 2012 Mar;129(3):548-56

Previous Research• Good reporting of some search elements2

– Database names, search terms

• Poor reporting of other elements2

– Date search executed, database platform

• Reproducibility is heterogeneously defined, but generally low3,4

2 Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54.3 Yoshii et al. J Med Libr Assoc. 2009 Jan;97(1):21-9.4 Golder et al. J Clin Epidemiol. 2013 Mar;66(3):253-60

Purpose

1. Examine search reporting and reproducibility in systematic reviews published in high-impact medicine journals in 2012

2. Investigate the impact of librarian involvement and reporting guidelines on reporting/reproducibility rates

Hypotheses

1. Reporting/reproducibility rates will be higher when a librarian was involved.

2. Reporting/reproducibility rates will be higher when a guideline was mentioned or required by the journal.

Journals Included

Cardiology• Circ Cardiovasc Genet• Circ Cardiovasc Interv• Circ Heart Fail• Circ Res• Circulation• Eur Heart J• J Am Coll Cardiol• JACC Cardiovasc Imaging• JACC Cardiovasc Interv• Nat Rev Cardiol

Surgery• Am J Surg Pathol• Am J Transplant• Ann Surg• Ann Surg Oncol• Arch Surg• Br J Surg• Endoscopy• J Am Coll Surg• J Neurol Neurosurg

Psychiatry• Surg Obes Relat Dis

Pediatrics

• Arch Dis Child• Arch Dis Child Fetal

Neonatal Ed• Arch Pediatr Adolesc Med• Eur Child Adolesc

Psychiatry• J Am Acad Child Adolesc

Psychiatry• J Pediatr• Pediatr Allergy Immunol• Pediatr Infect Dis J• Pediatrics• Semin Fetal Neonatal Med

Identification of Studies

• PubMed search on 9/2/2013 using modified Montori strategy5

(search*[Title/Abstract] OR meta-analysis[Publication Type] OR MEDLINE[Title/Abstract] OR EMBASE[Title/Abstract] OR Meta-analysis[Title/Abstract] OR (systematic[Title/Abstract] AND review[Title/Abstract]) AND (("2012/01/01"[PDat] : "2012/12/31"[PDat])

• PubMed SR subset filter also used for comparison– <4% of included articles found only by Montori strategy

5 Montori et al. BMJ. 2005 Jan 8;330(7482):68.

Identification of Studies

• All retrieved studies screened by both authors to identify systematic reviews– Searches at least one published literature database– Inclusion/exclusion criteria are prespecified – Does not limit to a certain number of journals or subset of

journals

• Disagreements resolved by review of original articles and consensus

Cardiology Surgery Pediatrics

216 317 201

51 142 77

Retrieved

Included

270

Items Identified

• Items covered:1. Search elements (eg., terms, strategy, limits)

2. Librarian involvement (author and role)

3. Guideline mentioned

Data Extraction

• Data extraction form created in Qualtrics• Included articles split between authors• Initial entry completed in March, 2014

Analyses

• SPSS used for all analyses– Logistic regression (odd ratio)– Chi Square/Fisher (p-value)– Mann-Whitney/T-Test (means)– α =.01

Search Updated

Search Execution Date

Full Strategy Named

Booleans

Start Year

Providers Named

Start/Last Year

Limits Indicated

Last Year

Specific Terms Named

Databases Searched

0% 25% 50% 75% 100%

3%

22%

34%

44%

60%

68%

73%

75%

81%

89%

100%

No significant differences based on discipline

Search Updated

Search Execution Date

Full Strategy Named

Booleans

Start Year

Providers Named

Start/Last Year

Limits Indicated

Last Year

Specific Terms Named

Databases Searched

0% 25% 50% 75% 100%

3%

22%

34%

44%

60%

68%

73%

75%

81%

89%

100%

0.16

0.31

0.6

0.67

0.76

0.98

Current StudySampson 2008*

*

*

*

*

* p<.01 for comparison

Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54.

Librarian Involvement

4%

12%

83%

1%Librarian Author

Librarian in Text/Acknowledgement

Librarian Not Men-tioned

Unknown

Cardiology Surgery Pediatrics0%

5%

10%

15%

20%

25%

30%

35%

40%

0%

13%

34%

P<.001P<.01

P<.001

49%51%

Reporting Guideline Mentioned

Guideline Men-tionedGuideline Not Men-tioned

84%

16%

PRISMA Flowchart Provided

FlowchartNo Flowchart

62%

38%

Journal Requires PRISMA

PRISMA RequiredPRISMA Not Required

Conservative Reproducibility

• Includes full search strategy for at least one database– Database searched– Provider– Date range– Search date– Terms– Booleans– Indication of limits

Liberal Reproducibility

• Includes core search elements– Database– Date range– Specific terms– Indication of limits

Reproducibility Rates

• Reproducibility rates <8% in prior studies– Previous criteria mirrored conservative criteria

Conservative Liberal

Surgery 11% 56%

Pediatrics 12% 57%

Cardiology 12% 39%

Total 11% 53%

Librarian Guideline Journal

Database Named ns ns ns

Databases Searched MD 1.34 (.78-1.89)*** ns ns

Provider Named ns ns ns

Search Years Indicated ns ns ns

Search Date ns ns ns

Search Updated ns ns ns

Search Terms OR .2 (.08-.49)*** ns ns

Full Strategy Presented ns ns ns

Boolean ns ns ns

Limits ns ns ns

Conservative Reproducibility

ns ns ns

Liberal Reproducibility ns ns ns

**p<.01 ***p<.001Discipline included in regression

Librarian Author/None Librarian Text/None

Database Named ns ns

Databases Searched MD 2.8 (1.8-3.9)*** MD 1.3 (.7-2.0)***

Provider Named n/a ns

Search Years Indicated ns ns

Search Date ns ns

Search Updated OR .08 (.01-.52)** ns

Search Terms OR .16 (.04-.71)** OR .18 (.07-.48)***

Full Strategy Presented ns ns

Boolean ns ns

Limits ns ns

Conservative Reproducibility

ns ns

Liberal Reproducibility ns ns

**p<.01 ***p<.001Discipline, Journal Requirements and Guideline Mentioned included in regression

Results

• Reproducibility is not predicted by:– Librarian involvement– Journal requirements– Guidelines mentioned in the paper

Results

• Librarian Involvement– Increases number of databases searched– Decreases updating of search results

when librarian is an author– Decreases reporting of search terms

Limitations

• Published information– Librarian could be involved, but not

mentioned• Ambiguous search elements

– Booleans? Limits? Full strategy?• Limited to three medical disciplines and

a single year

Next Steps

• Duplicate entry of data• Examination of reporting in greater

depth– Publication lag– Presentation of elements for more than

one database

Conclusions

1. Librarian involvement does not intrinsically mean better reporting

2. We must work with authors and publishers to improve reporting

Thank You

Questions?

Jonathan Koffel

jbkoffel@umn.edu