1
Review
Systematic Reviews and Meta-analysis
Trials
Steven Goodman, MD, PhD
Systematic Review
A review that has been prepared using a systematic approach to minimize biases and random errors.
- Use predefined and explicit methodology. - Review process: i) focused question, ii) search, iii) study selection, iv) quality appraisal, v) data extraction, vi) data analysis
Chalmers and Altman, 1994
Meta-analysis
A statistical technique for combining results of several studies into a single numerical estimate.
Also called statistical pooling.
Chalmers and Altman, 1994
Definitions cont’d
Therefore, you can have a systematic review without a meta-analysis, but you should NOT have a meta-analysis without a systematic review.
Why Perform a Systematic Review? To start an investigation To write a grant proposal To write a manuscript To write a thesis To develop or choose a health policy To talk on a topic (e.g. testify) To write a paper summarizing the
review
Who is doing them? Fellows Faculty EPCs Govt Agencies IOM Professional societies Guideline panels Insurance Companies Lawyers Everyone (except many who should!)
2
SRs I have known Agent Orange and cancer Thimerosal and autism ICDs and mortality IVF and birth outcomes Mammography and breast cancer Treatment of PTSD in Veterans Gene expression tests for breast
cancer prognosis
The Problem
Areas of concern Clinical practice, or justification of RCT’s, does
not reflect evidence that is already conclusive. Clinical practice has become entrenched and
clinical trials difficult, in spite of absent or poor evidence.
Literature reviews based on a non-comprehensive, non-systematic searching and evaluation of the literature, followed by inappropriate mathematical methods.
The case of aprotinin
The aprotinin story (Fergusson et al., Clinical Trials, 2005) Drug to reduce blood loss during surgery. 64 studies, 1987 and 2002, N=20 to 1784,
median 63. Efficacy established almost immediately, effect
stabilized by 1992, after 12 studies. Median of 20% of prior trials cited, only 7/44
(15%) trials cited the largest one. Systematic reviews in 1992 and 1997 were cited
only 3 times.
3
1/3 1
Aprotinin worse
Aprotinin better
Citation pattern for 64 Aprotinin studies
“The scandalous failure of science to cumulate evidence scientifically” Iain Chalmers, Clinical Trials, 2:12-14, 2005.
“The article by Dean Fergusson and his colleagues ... is the most recent evidence of an ongoing scandal in which research funders, academia, researchers, research ethics committees and scientific journals are all complicit. New research should not be designed or implemented without first assessing systematically what is known from existing research. The failure to conduct that assessment represents a lack of scientific self-discipline that results in an inexcusable waste of public resources. In applied fields like health care, failure to prepare scientifically defensible reviews of relevant animal and human data results not only in wasted resources but also in unnecessary suffering and premature death.”
Albumin Meta-analysis
4
Albumin meta-analysis reaction Albumin meta-analysis reaction
BMJ Editorial “Every 100 patients treated means six more deaths. If true, this is deeply
disturbing.”
London Observer “300 die as health chiefs dither.”
“Burns cure kills thousands”
Some breast cancer examples
Cochrane Final Summary: 13 Yr Total Mortality
Cochrane Summary: 13 Yr Breast Ca Mortality
5
Cochrane Summary: 13 Yr All Cancer Mortality
Cochrane Summary: Mastectomy + Lumpectomy
M+L Rate in Screening arm: 1.63% M+L Rate in No Screening arm: 2.15%
Difference = 0.5%
Cochrane Summary: Mastectomies alone
Problems with non-systematic reviews Noncomprehensive search and no
criteria for inclusion Only English language No methodological quality
assessment Don’t properly account for
quantitative weight of evidence (a lot of “vote counting”
Outdated Often opinion pieces parading as
objective evidence summaries
Meta-analytic principles • Different studies addressing the
same question are estimating approximately the same effect.
• Each study contributes to the net weight of evidence - they are not "positive" or "negative"
How to interpret a plot
Treatment better Control better
Point estimates+CI (from individual trials)
Pooled estimate+CI (meta-analysis)
1
1
2
3
6
A simple example of meta-analysis
Two studies looking at the 5 year mortality in subjects on Peritoneal Dialysis vs. Hemodialysis
A simple example of meta-analysis
Trial #1: N=160 PD 5 yr Mortality: 15% HD 5 yr Mortality: 25% Statistics: 10% difference, ±13%
p = 0.17, "Non-significant" Trial #2: N=300
PD 5 yr Mortality: 10% HD 5 yr Mortality: 15% Statistics: 5% difference, ±7%
p = 0.22, "Non-significant"
A simple example of meta-analysis
-5
0
5
10
15
20
25
Pooled Effect
PD better HD better
Steps in a meta-analysis • Assemble team of content and methods experts • Literature search • Select articles by defined criteria • Summarize qualitative design characteristics in "evidence
tables" • Assess and describe qualities of evidence • Decide whether studies can be pooled:
Qualitative and Quantitative criteria • Pool data if appropriate, w/ stat. model and measure of
effect • Conclusions, with appropriate caveats
Problems for all systematic reviews Publication Bias
Investigators are the main source Refers to both delay and non-publication
Qualitative heterogeneity Designs or patients are different enough so
Total evidence < S Indiv. study evidence Quantitative heterogeneity
Effect sizes that differ beyond what chance might produce.
Heterogeneity
7
Heterogeneity Evidence Tables
The Cochrane Collaboration What it does and how it does it...
The Cochrane Collaboration aims to help people make well-informed decisions about healthcare by
preparing, maintaining and promoting the accessibility of systematic reviews of the effects of
healthcare interventions
Archie Cochrane
1993 - First Cochrane Colloquium. Oxford, England
8
It’s about collaboration
13,000 active contributors 90 countries
Working together toward a common goal
The Cochrane Library
Published quarterly by John Wiley, Inc Cochrane Database of Systematic
Reviews (4677 reviews and protocols)
Other databases CENTRAL Register of Controlled Trials (489,167) Methodology Register (9048) Database of Methodological Reviews (19) Database of Abstracts of Reviews of Effectiveness (5931) NHS Health Technology Assessments (6358)
http://www.welch.jhu.edu/
9
So how is all this work organized?
www.cochrane.org
Steering Group
Centers Review Groups
Fields
Networks Methods Groups
Secretariat Cochrane Centers (12) and Branches (15)
Australasian Centre Thai Network, New Zealand Branch, Singapore Branch, South Asian Branch
Brazilian Centre Canadian Centre Chinese Centre
Hong Kong Branch Dutch Centre
Belgian Branch German Centre Iberoamerican Centre
Colombian Branch, Venezuelan Branch Italian Centre Nordic Centre
Finnish Branch, Norwegian Branch, Russian Branch South African Centre
Nigerian Branch UK Centre
Bahrain Branch US Center
Baltimore Center, Boston Branch, San Francisco Branch
Cochrane Review Groups (CRGs) (n=51)
Cochrane Acute Respiratory Infections Group Cochrane Airways Group Cochrane Anaesthesia Grop Cochrane Back Group Cochrane Bone, Joint and Muscle Trauma Group Cochrane Breast Cancer Group Cochrane Childhood Cancer Group Cochrane Colorectal Cancer Group Cochrane Consumers and Communication Group Cochrane Cystic Fibrosis and Genetic Disorders Group Cochrane Dementia and Cognitive Improvement Group Cochrane Depression, Anxiety and Neurosis Group
Cochrane Developmental, Psychosocial and Learning Problems Group Cochrane Drugs and Alcohol Group Cochrane Ear, Nose and Throat Disorders Group Cochrane Effective Practice and Organisation of Care Group Cochrane Epilepsy Group Cochrane Eyes and Vision Group Cochrane Fertility Regulation Group Cochrane Gynaecological Cancer Group……………
Cochrane Fields (n = 13) Behavioral Medicine Field Campbell and Cochrane Health Equity Field Cancer Network Child Health Field Complementary Medicine Field Health Care of Older People Field Health Promotion and Public Health Field Neurological Network Occupational Health Field Prehospital and Emergency Health Field Primary Health Care Field Rehabilitation and Related Therapies Field Vaccines Field
10
Methods groups (n = 11) Applicability and Recommendations Methods Group
Campbell and Cochrane Health Economics Methods Group
Individual Patient Data Meta-analysis Methods Group
Information Retrieval Methods Group
Non-Randomised Studies Methods Group
Patient Reported Outcomes Methods Group
Prospective Meta-Analysis Methods Group
Qualitative Methods Group
Bias Group
Screening and Diagnostic Tests Methods Group
Statistical Methods Group
Consumer Network
Base/Convenor in the US USCC (Baltimore is main contact); also San Francisco,
Boston Review Editorial bases and Satellites
Prostatic Diseases and Urologic Cancers (Minneapolis) HIV/AIDs/STDs (San Francisco) Neonatal (Burlington, Vt) Pain, Palliative and Supportive Care – Headache satellite
(Durham, NC) Eyes and Vision (US satellite) – (Baltimore)
Fields Complementary Medicine (Baltimore) Behavioral Medicine (New York) Primary Care (New York)
Methods Groups Screening and Diagnostic Tests (Providence, RI)
Consumers [Consumers United for Evidence-Based Healthcare
(Baltimore)]
Getting involved in the US Visit www.cochrane.org and www.cochrane.us Make sure your institution subscribes to The Cochrane
Library (CLib) Get into CLib, get to know how it works and how it can
help you and your work Attend the Colloquium (October 2007 in Sao Paolo) Subscribe to CCInfo Sign up for a workshop (how to do a systematic review,
what is evidence-based healthcare, handsearching, etc) Handsearch a journal Volunteer to be a peer reviewer Find the review group, field, methods group that best
meets your needs Co-author a review Get on the USCC’s contact list
11
2003 Barcelona – Iberoamerican Center staff serenade 2003 Barcelona – Gaudi welcomes Cochrane
Info and Contact
www.cochrane.org www.cochrane.us [email protected]