Assessing study quality for a systematic review

Trudy Bekkering, PhDCenter of Evidence-Based Medicine &

Belgian Branch of the Dutch Cochrane CenterCentre for Methodology of Educational Research

Katholieke Universiteit Leuven, Belgium

Why is it so important?

• Meta-analysis aims to increase precision• Meta-analysis of studies with bias in results

gives very precise but wrong results• Garbage in, garbage out

Bias versus imprecision

Bias:• A systematic error in the results or the

inferences• Methodological flaw • Overestimation or underestimation

Bias versus imprecision

BIAS Ideal study

Bias versus imprecision

Imprecision:• A random error in the results• Sample variation • Direction of error is random

Bias versus imprecision

Ideal study IMPRECISION

Bias versus imprecision

BIAS + IMPRECISION

Bias versus imprecision

BIAS ? IMPRECISION?

Risk of bias versus bias

• Clear empirical evidence that particular flaws in study design can lead to bias.

• Usually impossible to know to what extent biases have affected the results.

• Key consideration = should the results be believed

Tools for assessing study quality

Scales: discouragedChecklist: between 3 and 57 itemsCochrane tool: “domain based”

Depends on study design

Tool for RCTs: Cochrane tool

Risk of bias on 6 domaines:

1. Sequence generation

2. Allocation concealment

3. Blinding

4. Incomplete outcome data

5. Selective reporting

6. Other sources of bias

Risk of which biases?

Selection bias Differences between baseline characteristics of the groups compared

Performance bias Differences in the care that is provided, or in exposure to other factors than the intervention

Attrition bias Differences in withdrawals from a study

Detection bias Differences in how outcomes are determined

Reporting bias Differences between reported and unreported outcomes

How do you assess?

Domaine Description Judgement

Sequence generationQUOTE: “patients were randomly allocated”COMMENT: probably done, since earlier reports of this study describe use of random sequences

YES (low risk of bias)

Allocation concealmentYES (low risk)NO (high risk)UNCLEAR (uncertain)

BlindingYESNOUNCLEAR

Incomplete outcome dataYESNOUNCLEAR

Selective outcome reporting

YESNOUNCLEAR

Other sources of biasYESNOUNCLEAR

Adequate Not adequate Unclear Random number table

Computer generated list

Coin tossing Shuffling cards /

envelopes minimization

Generated by: Odd/even date of birth Date/day of admission Hospital record number

Allocation by: Clinical judgement Participant’s preference Result of lab test Availability of intervention

If insufficient information about sequence generation!

Sequence generation

Allocation concealment

Adequate Not adequate Unclear Central randomisation

(telephone, web-based, pharmacy)

Sequentially numbered drug containers of identical appearance

Sequentially numbered, opaque, sealed envelopes

Open random allocation schedule

Envelopes without appropriate safeguards

Quasi-randomisation

If insufficient information about sequence generation!

(e.g. “sealed envelopes”)

Blinding of intervention

• Participants (patients, clients)

• Care providers (doctors, nurses, teachers …)

• Outcome assessors

Blinding of intervention

Adequate Not adequate Unclear No blinding, outcome

(assessment) not likely to be influenced by lack of blinding

Blinding ensured and unlikely to have been broken

Either participants or some personnel unblinded, but unlikely to introduce bias + outcome assessment blinded

No or incomplete blinding and outcome (assessment) likely to be influenced by lack of blinding

Blinding attempted but likely that could have been broken

insufficient information

Issue not addressed in the study

Incomplete outcome data

“Attrition” (drop-out): no data

• Withdrawal• Do not attend follow-up appointment• Failure to complete questionnaire / diaries• Cannot be located (lost to follow-up)• Decision by investigator to cease follow-up• Data or records are lost

Incomplete outcome data

“Exclusion”: data available, but excluded from analysis

• Participants found to be ineligible after enrolment

• An “as treated” (or per-protocol) analysis: participants are only included if they received the intended intervention in accordance with the protocol

Assessing risk of bias

Low risk of bias High risk of bias

Complete outcome data

Missing in both groups but reasons are reported and balanced across groups

Reason unlikely to be connected with outcome (moved away)

Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effects in experimental group)

Differences in the reasons for missing data (e.g. smoking cessation)

“as treated” (per protocol) analysis

Selective outcome reporting

= selection of a subset of the variables recorded for inclusion in

publication,

on the basis of the results

For example:

• Omission of non-significant outcomes

• Choice of data for an outcome (e.g. osteoporosis)

• Choice of analysis (e.g. blood pressure)

• Reporting of subsets of data (e.g. sepsis)

• Under-reporting of data (e.g. only “not significant”)

Presentation in your review

“Risk of bias graph”

“Risk of bias summary”

Assessing risk of bias in NRS

•Selection bias (how was group allocation?)

•Performance bias (blinding, fidelity of interventions)

•Attrition bias (completeness of sample & follow-up)

•Reporting bias (selective outcome reporting)

•Confounding and adjustment

Confounding

Comparison intervention - control

Intervention versus control

Difference in outcome

Imbalance in prognostic factors

?

?

Confounding

Association between 2 factors

Presence of risk factor

Occurrence of outcome

Confounding factor

Confounding & adjustment

• At the stage of protocol: list potential confounding factors

• Identify the factors the authors have considered and omitted

• Assess balance between groups at baseline

• What did authors do to control for confounders (matching, restricting to subgroups, stratification, regression modelling)

Tool for NRS

Downs and Black instrument (J Epidemiol Community Health 1998;52:377-84)

27 items:• Reporting (10)• External validity (applicability) (3)• Internal validity - bias (7)• Internal validity – confounding (6)• Power (1)

Downs and Black instrument

http://www.nccmt.ca/ registry/view/eng/9.html

Tool for NRS

Newcastle-Ottawa Scale (NOS)

(Wells 2008)

8 items covering 3 perspectives:

•Selection of study groups

•Comparibility of groups

•Ascertainement of exposure (case-control) or outcome (cohort)

http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

Diagnostic studies

QUADAS toolWhiting BMC Medical Research Methodology 2003;3:25

Cochrane version: 11 items (out of 14 original)

Diagnostic Test Accuracy Working Group: handbook

http://srdta.cochrane.org/handbook-dta-reviews

Other risk of bias assessment tools

SIGN (Scottish Intercollegiate Guidelines Network)

http://www.sign.ac.uk/methodology/checklists.html

In summary

• Risk of bias assessment is essential for systematic reviews

• For RCT: use the Cochrane tool• For NRS:

• Higher risk of bias (selection bias & reporting bias)• Use the appropriate tool to assess risk of bias• Consider how potential confounders are addressed