centre for evidence based intervention

Centre for Evidence Based Intervention

Evidence from Review Groups

Objectives

Identify policies & procedures on…

Determining publication of empty reviews

Practice of handling excluded studies across Cochrane

and other providers

Informing practice based on empty reviews and

excluded studies

Methods

Survey with closed and open-ended questions

53 Cochrane Collaboration Review Groups

WHO, JBI, What Works Clearinghouse and SCIE

Coded & analyzed responses

Why might a review be empty?

“because it is a new or recently introduced intervention

which has yet to gather strength of evidence”

“the intervention does not lend itself to randomised

controlled trials”

“poor question?”

……..

Cochrane Groups publishing empty reviews?

N=53

Rationale for publishing Empty Reviews

Page 6

N=46

Other providers

WHO - doesn't have a policy on empty reviews. In a way it isn't relevant - they would search for the best available evidence, but would need to make a recommendation regardless of what they find.

SCIE - As the evidence base in social care is very limited, we may often find that a review with inclusion criteria specifying high quality controlled (and/or randomised) studies of specific interventions would be ‘empty’ – hence we would not commission a review.

WWCH - review focus is changed depending on the evidence base, thus empty reviews are not generally produced.

Views about including empty reviews Include to instigate further research

“Our aim with empty reviews is to raise the profile of the question and hopefully generate research leading to high quality evidence which we can then report quickly in an updated review.”

Include if good case made “There may be benefit in amending the TRF form to include

the line: ‘If you think this review might be empty, please give your reasons why you feel it is important to do the review.’”

Reasons for not publishing empty reviews

(Some groups have minimum # of studies stipulation - e.g., 2-3)Not worth time and resources

“a colossal waste of everyone’s time and energy”

Influence on group’s impact factorMay be misleading - Empty reviews (< 3 studies) can put positive supportive evidence from initial, low quality studies behind treatmentBUT, question can still matter

“may be exceptions where review questions appear of great public health relevance”

Handling excluded studies

Standard policy? Permitted in review?

Conditions for providing evidence based on excluded studies N=31

Further considerations for whether to allow excluded studies Excluded based on intervention or outcomes?

RCTs feasible/ethical for topic area?

If not, lower grade studies may be acceptable

Guidelines may be needed for application of evidence from

excluded studies

“We dissuade authors from providing evidence from excluded studies

because of inconsistency, which would lead to provision of evidence

that is arbitrary and unsystematic.”

Guidance for ‘Implications for practice’ section

For ‘Yes’ responses:

(n=7)

Extra guidance but on case-by-case

basis (3)

General advice (2)

Refer to other empty reviews, EPICOT,

& extra advice (1)

Group Methodologists give guidance (1)N=53

Aim to update or withdraw empty

reviews faster than others?

For ‘Yes’ responses:

(n=4)

No reason given (2)

If there are recently published RCTs

(2)N=53

Views about priority on updating empty reviews

No, topic-based priority only “Priority is only given on the basis of importance of the review topic, not on

whether there are included studies or not.”

Only if new study emerges “Aim to update/review them around the same time as other reviews , but if

we became aware of a study relevant to an empty review, it would probably prompt a more rapid update.”

Thank you!

paul.montgomery@spi.ox.ac.uk