critical appraisal of the literatureacademic.sun.ac.za/stellmed.v2/coursematerial/leadership...
TRANSCRIPT
CRITICAL APPRAISAL OF THE LITERATURE
Prof AE Zemlin
Division of Chemical Pathology
Tygerberg Hospital
Introduction
• Part of EBM
• Research should be reported transparently:
• What was planned?
• What was done?
• What was found?
• What conclusions?
Evidence-based decisions
Part of EBLM
ASK
ACQUIRE
APPRAISE
ANALYSE
APPLY
ASSESS
EVIDENCE IN LABORATORY MEDICINE
Systematic review
• Benefits:• Multiple studies
• Meets statistical power
• Can identify variations in practice
• Limitations:• Often few studies meet inclusion and quality criteria
• Require identical outcome measures
Randomized control trial*
• Benefits:• Offers lowest risk of bias
• Limitations:• Assessment of test and treat intervention
• Generalizability to entire population for test complicated
• Blinding difficult
Observational study, e.g. audit
• Benefits:• Most representative of routine practice
• Useful for audit
• Good for assessing care processes, e.g. laboratory practice guidelines
• Limitations:• Intervention and control groups may not be rigorously
matched
• Open to bias
Before and after trial
• Benefits:• Practical design
• Limitations:• Risk of operator or incorporation bias
Case-control study
• Benefits:• Adds insight for exploratory questions
• Limitations:• Exploratory information only
• May not hold in prospective studies
Case study
• Benefits:• Provides information for infrequently encountered
patients
• Limitations:• Limited information and likely not evidence based as
underpowered
Expert opinion
• Benefits:• Insight into how knowledgeable experts approach
questions / problems
• Limitations:• Biases of expert
BIAS – DEPARTURE FROM TRUENESS
Spectrum bias
• Performance of test may differ in different patient populations
• Found often in case-control studies – 2 populations with different characteristics
• Extreme – healthy vs disease
• Greatest effect on results
• Also called selection / recruitment bias
Underpowered study bias
• Power calculations important part of study design
• More for RCT on therapeutic interventions
• Consequence – no insight into variability of measure
Attrition bias
• Participants lost to follow-up
• Must be accounted for
Verification bias
• Work-up bias of referral bias
• Results of index test influence choice of reference test
Observer / operator bias
• Person using test is aware of other info about patient or other investigations
• Review bias – can lead to overestimation of effect
• Can be mitigated by incorporating blinding into study
Disease progression bias
• Long delay between index and reference test being performed
• Change in disease status
Performance bias
• Change in performance
Test reproducibility bias
• Test reproducibility – impact on clinical performance and effectiveness
• Technical performance of analytical methods needs to be described in detail
Reporting bias
• Prevent poor diagnostic studies being reported
• STARD - checklistsm
Publication bias
• Reluctance to publish negative data
• Meta-analysis may not be representative
Conducting a critical appraisal
• Checklist
• Identify problem
• PICO
• EQUATOR – international initiative seeks to improve the value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines
EQUATOR – Enhancing the Quality and Transparency Of health Research
• Diagnostic / prognostic studies – STARD
• Randomised trials – CONSORT
• Observational studies – STROBE
• Systematic reviews – PRISMA
• Case reports – CARE
• Quality improvement studies – SQUIRE
Minimum information that should be included
Standards for Reporting of Diagnostic Accuracy Studies (STARD)
• Completeness and transparency
• Minimum essential items
• Judge for potential bias
Title / Abstract
• Identification as a study of diagnostic accuracy using at least one measure of accuracy
• Structured summary of study design, methods, results and conclusions*
Introduction
• Scientific / clinical background including intended use and clinical role of index test *
• Study objectives and hypothesis *
Methods – study design
• Was data collection planned before index test and reference standard were performed (prospective) of after (retrospective)?
Methods - participants
• Eligibility criteria
• How were they identified?
• Where and when were potentially eligible participants identified?
• Did participants form a consecutive, random or convenience series?
Methods – test methods
• Index test
• Reference tests
• Rationale for choosing reference standard
• Definition and rationale for test positivity cut-offs or result categories of reference standard
• Definition and rationale for test positivity cut-offs or result categories of index test
• Whether clinical information and reference standard results were available to the performers / readers of the index test
• Whether clinical information and index test results were available to the assessors of reference standard
Methods -analysis
• Methods for estimating or comparing measures of diagnostic accuracy
• How indeterminate index test or reference standard results were handled
• How missing data on the index test or reference standard were handled
• Any analyses of variability in diagnostic accuracy, distinguishing pre-specified from exploratory
• Intended sample size and how was determined *
Results - participants
• Flow diagram of participants
• Baseline demographic and clinical characteristics of participants
• Distribution of severity of disease in those with target condition
• Distribution of alternative diagnoses in those without target condition
• Time interval and any clinical interventions between index test and reference standard
Results – test results
• Cross tabulation of the index test results (or their distribution) by the results of the reference standard
• Estimates of diagnostic accuracy and their precision (e.g. 95% CI)
• Any adverse events from performing index test or reference standard
Discussion
• Study limitations, including sources of potential bias, statistical uncertainty, and generalisability *
• Implications for practice, including the intended use and clinical role of the index test *
Other information
• Registration number and name of registry *
• Where full study protocol can be assessed *
• Sources of funding and other support; role of funders *
Conclusion
• Necessary to assess quality and relevance of evidence to support clinical decision and policymaking
• Number of checklists available depending on type of study being appraised
• All assess:1. Question being addressed
2. Quality of methodology used
3. Quality of results
4. Applicability to the problem and potential consequences of using this evidence
• Failure to adequately appraise may lead to inappropriate testing
QUESTIONS??