systema(c*reviews,*riskof*bias andevidenceevaluaon
TRANSCRIPT
![Page 1: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/1.jpg)
Tracey J. Woodruff, PhD, MPH Professor and Director
UCSF Program on ReproducBve Health and the Environment
Risk Assessment Specialty Sec1on (RASS)
January 14, 2015
Systema(c Reviews, Risk of Bias and Evidence Evalua(on for Characteriza(on of
Environmental Chemicals
![Page 2: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/2.jpg)
![Page 3: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/3.jpg)
3
(CUE workshop has a good cartoon)
![Page 4: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/4.jpg)
Environmental health literature is vast, diverse, and of variable quality
How do we evaluate scien0fic evidence to
make decisions? And shorten the 0me between science and
decision?
![Page 5: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/5.jpg)
5
Clinical sciences have faced and addressed these same challenges
Thank you Lisa Bero
![Page 6: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/6.jpg)
6
What is Evidence Based Medicine?
• Defined as, the “conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of
individual patients” ���
• In general, systems of EBM combine:
(1) existing scientific evidence;
(2) clinical expertise; and
(3) patient values and preferences
to make diagnosis, prognosis and treatment decisions.
![Page 7: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/7.jpg)
7
Grades of Recommendation Assessment, Development
and Evaluation
![Page 8: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/8.jpg)
8
But Evidence Based Medicine
Methodologies Are Not Directly
Transferable to Environmental Science!
8
![Page 9: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/9.jpg)
Naviga1on Guide Methodology
A systemaBc and transparent method to evaluate the quality of evidence and to support evidence-‐based decision making, bridging the gap between clinical
and environmental health
BRIDGING CLINICAL & ENVIRONMENTAL HEALTH
Developed by UCSF’s Program on ReproducBve Health and the Environment in collaboraBon with the NavigaBon Guide Working Group in 2009
![Page 10: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/10.jpg)
Naviga1on Guide Work Group
The Cochrane Collabora1on
![Page 11: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/11.jpg)
Overview of the Methodology
11
![Page 12: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/12.jpg)
“ …systema1c-‐review standards provide an approach that would substanBally strengthen the IRIS process…” NAS 2014
“EPA should consistently use a more systema1c approach to evaluaBng the literature ……….” NAS 2014
12
![Page 13: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/13.jpg)
13
Weight of Evidence
“too vague and is of lidle scienBfic use”
NaBonal Academy of Sciences 2014
![Page 14: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/14.jpg)
![Page 15: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/15.jpg)
Evaluate each evidence stream separately using systemaBc and transparent approaches
![Page 16: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/16.jpg)
Systema1c Review Approach for Each Evidence Stream
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate Strength of Evidence Non Human
Data
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Human Data
Overall Co
nclusio
n
A pre-‐specific analy1c plan (protocol) is developed and applied consistently to the evidence.
![Page 17: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/17.jpg)
Systema1c Review Approach for Each Evidence Stream
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data
Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Overall Co
nclusio
n
A pre-‐specific analy1c plan (protocol) is developed and applied consistently to the evidence.
opula1on xposure omparator utcome
![Page 18: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/18.jpg)
“PECO” Statement
SystemaBc search
Select Studies
Extract Data & Data Analysis
Rate Quality of Evidence
Rate the Strength of Evidence
Rate Quality of Evidence
Rate Strength of Evidence
Rate the Quality and Strength of the Evidence
High Moderate
Low
Sufficient evidence of toxicity
Limited evidence of toxicity
Inadequate evidence of toxicity
Evidence of lack of toxicity
![Page 19: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/19.jpg)
• Risk of Bias
• RaBng quality of evidence
Rate Quality of Evidence
![Page 20: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/20.jpg)
Lam et al. EHP 2014
![Page 21: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/21.jpg)
True Effect 0
True Effect 0
Risk of Bias vs Random Error 1. Bias 2. Random Error
![Page 22: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/22.jpg)
“The commidee notes that assessing the quality of the study is not equivalent to assessing the risk of bias in the study. An assessment of study quality evaluates the extent to which the researchers conducted their research to the highest possible standards and how a study is reported. Risk of bias is related to the internal validity of a study and reflects study-‐design characterisBcs that can introduce a systemaBc error (or deviaBon from the true effect) that might affect the magnitude and even the direcBon of the apparent effect. An assessment of risk of bias is a key element in systemaBc-‐review standards; potenBal biases must be assessed to determine how confidently conclusions can be drawn from the data.” The commidee emphasizes the importance of assessing risk of bias for all study types.
“the commidee emphasizes the need for EPA to assess the “risk of bias” in individual studies.” [for all study types]
NaBonal Academy of Sciences 2014
Risk of Bias & NAS
![Page 23: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/23.jpg)
Low risk
Probably low risk
Probably high risk
High risk
Sequence generation Allocation concealment Blinding Incomplete outcome data Selective reporting
Other bias Conflict of interest
0% 20% 40% 60% 80% 100%
Recruitment strategy Blinding Confounding Exposure assessment Incomplete outcome data Selective reporting Conflict of interest Other bias
Evaluate Risk of Bias
![Page 24: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/24.jpg)
Factors for downgrading/upgrading evidence were derived directly from factors used in GRADE and Cochrane
High
Moderate
Low
Animal evidence • Level of study control • RandomizaBon key factor in default of “high” in GRADE
• Studies find humans more suscepBble to chemical exposures than animals
Human evidence
• Non-‐randomized, but variable quality. Cochrane review finds similar results between RCTs and ObservaBonal studies
Rate the Quality of the Evidence
![Page 25: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/25.jpg)
Factors that DECREASE quality level
1. Risk of bias (study limita1ons)
Possible raBngs: 0=no change; -‐1 or -‐2 downgrade 1 or 2 levels For Human and Animal evidence
2. Indirectness
3. Inconsistency
4. Imprecision
5. Publica1on bias
![Page 26: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/26.jpg)
Factors that INCREASE quality level
1. Large magnitude of effect
2. Dose response
3. Confounding minimizes effect
Possible raBngs: 0=no change; -‐1 or -‐2 downgrade 1 or 2 levels For Human evidence only
![Page 27: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/27.jpg)
NaBonal Academy of Sciences 2014
![Page 28: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/28.jpg)
28
Evidence Stream
Human Non-‐human mammalian
StarBng raBng Moderate High
Downgrade
Risk of Bias 0 -‐1 Indirectness 0 0 Inconsistency 0 0 Imprecision 0 0 PublicaBon bias 0 0
Upgrade
Large magnitude effect 0 N/A Dose response 0 N/A All possible confounding would confirm negaBve result
0 N/A
Grade 0 -‐1
Final ra1ng Moderate Moderate
Summary of Quality of Evidence for PFOA
![Page 29: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/29.jpg)
QUALITY OF EVIDENCE
Moderate
High
Low
STRENGTH OF EVIDENCE (LEVEL OF CERTAINTY REGARDING TOXICITY)
Sufficient evidence of toxicity
Limited evidence of toxicity
Inadequate evidence of toxicity
Evidence of lack of toxicity
CRITERIA: 1. Quality of evidence: 2. What is the direc1on of effect? 3. What is the confidence in the
effect? 4. Are there other compelling
adributes of the data that influence certainty?
Rate the Strength of Evidence
![Page 30: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/30.jpg)
Methodological Needs • Criteria for moving from quality to strength of evidence
• Methods to include all potenBal types of evidence, i.e., assessing chickens, flies and in vitro data
• Improved methods of animal toxicity tesBng – high ROB may be prevalent for key domains
• MechanisBc data is considered under other consideraBons…. Further development needed
• Consider the nature and extent of consensus that is needed for a decision
![Page 31: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/31.jpg)
Conclusion
• Doable • Rigorous, systema1c, transparent and doable • Capacity to evolve with changes in evidence stream
![Page 32: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/32.jpg)
Acknowledgements PFOA Case Study Authors
![Page 33: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/33.jpg)
Acknowledgements Navigation Guide Funders (since 2009)
– USEPA – US Environmental Protection Agency STAR (RD83467801) – National Institute for Environmental Health Sciences (ES018135) – Cal-EPA – Fred Gellert Foundation – Clarence Heller Foundation – New York Community Trust – Forsythia Foundation – Passport Foundation – Johnson Family Foundation – Heinz Endowments – Rose Foundation – Kaiser Permanente – Planned Parenthood Federation of America – UCSF Phillip R Lee Institute for Health Policy Studies
![Page 34: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/34.jpg)
Program on ReproducBve Health and the Environment
Thank you
![Page 35: Systema(c*Reviews,*Riskof*Bias andEvidenceEvaluaon](https://reader031.vdocument.in/reader031/viewer/2022012419/6173ee044b3f690f900ec19c/html5/thumbnails/35.jpg)
Sir AusBn Bradford Hill -‐ incompleteness of science … “does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the ac0on that it appears to demand at a given 0me” (Hill 1965).