assessing study quality for a systematic review
DESCRIPTION
Assessing study quality for a systematic review. Trudy Bekkering , PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium. Why is it so important?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/1.jpg)
Assessing study quality for a systematic review
Trudy Bekkering, PhDCenter of Evidence-Based Medicine &
Belgian Branch of the Dutch Cochrane CenterCentre for Methodology of Educational Research
Katholieke Universiteit Leuven, Belgium
![Page 2: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/2.jpg)
Why is it so important?
• Meta-analysis aims to increase precision• Meta-analysis of studies with bias in results
gives very precise but wrong results• Garbage in, garbage out
![Page 3: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/3.jpg)
Bias versus imprecision
Bias:• A systematic error in the results or the
inferences• Methodological flaw • Overestimation or underestimation
![Page 4: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/4.jpg)
Bias versus imprecision
BIAS Ideal study
![Page 5: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/5.jpg)
Bias versus imprecision
Imprecision:• A random error in the results• Sample variation • Direction of error is random
![Page 6: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/6.jpg)
Bias versus imprecision
Ideal study IMPRECISION
![Page 7: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/7.jpg)
Bias versus imprecision
BIAS + IMPRECISION
![Page 8: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/8.jpg)
Bias versus imprecision
BIAS ? IMPRECISION?
![Page 9: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/9.jpg)
Risk of bias versus bias• Clear empirical evidence that particular flaws
in study design can lead to bias.• Usually impossible to know to what extent
biases have affected the results.• Key consideration = should the results be
believed
![Page 10: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/10.jpg)
Tools for assessing study quality
Scales: discouragedChecklist: between 3 and 57 itemsCochrane tool: “domain based”
Depends on study design
![Page 11: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/11.jpg)
Tool for RCTs: Cochrane tool
Risk of bias on 6 domaines:
1. Sequence generation
2. Allocation concealment
3. Blinding
4. Incomplete outcome data
5. Selective reporting
6. Other sources of bias
![Page 12: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/12.jpg)
Risk of which biases?
Selection bias Differences between baseline characteristics of the groups compared
Performance bias Differences in the care that is provided, or in exposure to other factors than the intervention
Attrition bias Differences in withdrawals from a study
Detection bias Differences in how outcomes are determined
Reporting bias Differences between reported and unreported outcomes
![Page 13: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/13.jpg)
How do you assess?
![Page 14: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/14.jpg)
Domaine Description Judgement
Sequence generationQUOTE: “patients were randomly allocated”COMMENT: probably done, since earlier reports of this study describe use of random sequences
YES (low risk of bias)
Allocation concealmentYES (low risk)NO (high risk)UNCLEAR (uncertain)
BlindingYESNOUNCLEAR
Incomplete outcome dataYESNOUNCLEAR
Selective outcome reporting
YESNOUNCLEAR
Other sources of biasYESNOUNCLEAR
![Page 15: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/15.jpg)
Adequate Not adequate Unclear Random number table
Computer generated list
Coin tossing Shuffling cards /
envelopes minimization
Generated by: Odd/even date of birth Date/day of admission Hospital record number
Allocation by: Clinical judgement Participant’s preference Result of lab test Availability of intervention
If insufficient information about sequence generation!
Sequence generation
![Page 16: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/16.jpg)
Allocation concealment
Adequate Not adequate Unclear Central randomisation
(telephone, web-based, pharmacy)
Sequentially numbered drug containers of identical appearance
Sequentially numbered, opaque, sealed envelopes
Open random allocation schedule
Envelopes without appropriate safeguards
Quasi-randomisation
If insufficient information about sequence generation!
(e.g. “sealed envelopes”)
![Page 17: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/17.jpg)
Blinding of intervention
• Participants (patients, clients)
• Care providers (doctors, nurses, teachers …)
• Outcome assessors
![Page 18: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/18.jpg)
Blinding of intervention
Adequate Not adequate Unclear No blinding, outcome
(assessment) not likely to be influenced by lack of blinding
Blinding ensured and unlikely to have been broken
Either participants or some personnel unblinded, but unlikely to introduce bias + outcome assessment blinded
No or incomplete blinding and outcome (assessment) likely to be influenced by lack of blinding
Blinding attempted but likely that could have been broken
insufficient information
Issue not addressed in the study
![Page 19: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/19.jpg)
Incomplete outcome data
“Attrition” (drop-out): no data
• Withdrawal• Do not attend follow-up appointment• Failure to complete questionnaire / diaries• Cannot be located (lost to follow-up)• Decision by investigator to cease follow-up• Data or records are lost
![Page 20: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/20.jpg)
Incomplete outcome data
“Exclusion”: data available, but excluded from analysis
• Participants found to be ineligible after enrolment
• An “as treated” (or per-protocol) analysis: participants are only included if they received the intended intervention in accordance with the protocol
![Page 21: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/21.jpg)
Assessing risk of bias
Low risk of bias High risk of bias
Complete outcome data
Missing in both groups but reasons are reported and balanced across groups
Reason unlikely to be connected with outcome (moved away)
Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effects in experimental group)
Differences in the reasons for missing data (e.g. smoking cessation)
“as treated” (per protocol) analysis
![Page 22: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/22.jpg)
Selective outcome reporting
= selection of a subset of the variables recorded for inclusion in publication,on the basis of the results
For example:
• Omission of non-significant outcomes
• Choice of data for an outcome (e.g. osteoporosis)
• Choice of analysis (e.g. blood pressure)
• Reporting of subsets of data (e.g. sepsis)
• Under-reporting of data (e.g. only “not significant”)
![Page 23: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/23.jpg)
Presentation in your review
“Risk of bias graph”
![Page 24: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/24.jpg)
“Risk of bias summary”
![Page 25: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/25.jpg)
Assessing risk of bias in NRS
•Selection bias (how was group allocation?)
•Performance bias (blinding, fidelity of interventions)
•Attrition bias (completeness of sample & follow-up)
•Reporting bias (selective outcome reporting)
•Confounding and adjustment
![Page 26: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/26.jpg)
Confounding
Comparison intervention - control
Intervention versus control
Difference in outcome
Imbalance in prognostic factors
?
?
![Page 27: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/27.jpg)
Confounding
Association between 2 factors
Presence of risk factor
Occurrence of outcome
Confounding factor
![Page 28: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/28.jpg)
Confounding & adjustment
• At the stage of protocol: list potential confounding factors
• Identify the factors the authors have considered and omitted
• Assess balance between groups at baseline
• What did authors do to control for confounders (matching, restricting to subgroups, stratification, regression modelling)
![Page 29: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/29.jpg)
Tool for NRS
Downs and Black instrument (J Epidemiol Community Health 1998;52:377-84)
27 items:• Reporting (10)• External validity (applicability) (3)• Internal validity - bias (7)• Internal validity – confounding (6)• Power (1)
![Page 30: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/30.jpg)
Downs and Black instrument
http://www.nccmt.ca/ registry/view/eng/9.html
![Page 31: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/31.jpg)
Tool for NRS
Newcastle-Ottawa Scale (NOS)(Wells 2008)
8 items covering 3 perspectives:•Selection of study groups•Comparibility of groups•Ascertainement of exposure (case-control) or
outcome (cohort)
![Page 32: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/32.jpg)
http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
![Page 33: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/33.jpg)
Diagnostic studies
QUADAS toolWhiting BMC Medical Research Methodology 2003;3:25
Cochrane version: 11 items (out of 14 original)
Diagnostic Test Accuracy Working Group: handbook
http://srdta.cochrane.org/handbook-dta-reviews
![Page 34: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/34.jpg)
![Page 35: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/35.jpg)
Other risk of bias assessment tools
SIGN (Scottish Intercollegiate Guidelines Network)
http://www.sign.ac.uk/methodology/checklists.html
![Page 36: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/36.jpg)
![Page 37: Assessing study quality for a systematic review](https://reader035.vdocument.in/reader035/viewer/2022070422/568164ac550346895dd6b024/html5/thumbnails/37.jpg)
In summary
• Risk of bias assessment is essential for systematic reviews
• For RCT: use the Cochrane tool• For NRS:
• Higher risk of bias (selection bias & reporting bias)• Use the appropriate tool to assess risk of bias• Consider how potential confounders are addressed