systematic review and meta-analysis for translating ... · systematic review and meta-analysis for...

Systematic Review and Meta-Analysis for Translating Toxicology into Better Health

Outcomes:the Navigation Guide Systematic Review Methodology

33rd International Symposium on Systematic Review and Meta--Analysis of Laboratory Animal StudiesWashington DC, November 14, 2014

Tracey J. Woodruff, PhD, MPH , Professor Program on Reproductive Health and the Environment

School of Medicine, University of California San Francisco

Federal reserve data on chemical production is only offered as relative production, which is unit-less. A specific reference year is chosen and values are calculated relative to that years production. In this particular data set 2007 is the reference year and is assigned a value of 100.

Data from: U.S. Federal Reserve Board, Division of Research and Statistics

↑15 fold

Majority of Chemicals Not Tested Before Entering the Market

minocyclinePerfluorooctanoic acid

Manufactured pharmaceuticals have to have safety data before use

Manufactured chemicals do NOT

Environmental health literature is

vast, diverse, and of variable quality

How do we evaluate scientific evidence to make decisions?

And shorten the time between science and decision?

Navigation Guide Methodology

A systematic and transparent method to evaluate the quality of evidence and to support evidence-based decision making, bridging the gap between clinical

and environmental health

BRIDGING CLINICAL & ENVIRONMENTAL HEALTH

Developed by UCSF’s Program on Reproductive Health and the Environment in collaboration with the Navigation Guide Working Group in 2009

Navigation Guide Work Group

The Cochrane Collaboration

Overview of the Methodology

10

“ …systematic-review standards provide an approach that would substantially strengthen the IRIS process…” NAS 2014

“EPA should consistently use a more systematic approach to evaluating the literature ……….” NAS 2014

11

Systematic Review Approach for Each Evidence Stream

“PECO” Statement

Systematic search

Select Studies

Extract Data & Data

Analysis

Rate Quality of Evidence

Rate Strength of EvidenceNon Human

Data


Systematic search

Select Studies

Extract Data & Data

Analysis


Rate the Strength of Evidence

Human Data

Ove

rall

Co

ncl

usi

on

A pre-specific analytic plan (protocol) is developed and applied consistently to the evidence.

Using the Navigation Guide Systematic Review Methodology

Does Perfluorooctanoic Acid (PFOA) affect fetal growth?

14Woodruff, Zota, Schwartz EHP 2011

99% Had PFCs

There were inconsistent associations reported for several different birth outcomes,

including birth weight, birth length, head circumference, and ponderal index, among the

five general population studies that measured PFOS and PFOA in the study subjects.

Olsen GW, Butenhoff JL, & Zobel LR (2009) Perfluoroalkyl chemicals and human fetal development: An epidemiologic review with clinical and toxicological

perspectives. Reproductive Toxicology 27(3–4):212-30.

Steenland K, Fletcher T, & Savitz DA (2010) Epidemiologic evidence on the health effects of perfluorooctanoic acid (PFOA). Environ Health Perspect

118(8):1100-8.

Cumulatively, the studies provide inconsistent suggestions of a possible decrement in

birth weight associated with PFOA exposure, with studies varying in whether the

association with PFOS is similar (Apelberg et al. 2007), stronger (Stein et al. 2009;

Washino et al. 2009), or weaker (Fei et al. 2009; Hamm et al. 2009) than that reported

for PFOA.

What do the reviews say?

For Birthweight?

Animals from non-human species that are studied during reproductive/developmental time period (before and/or during pregnancy for females or during development for embryos).

One or more oral, subcutaneous or other treatment(s) of any dosage with perfluorooctanoic acid (PFOA), CAS# 335-67-1, or its salts during the time before pregnancy and/or during pregnancy for females or directly to embryos.

Experimental animals receiving different doses of PFOA or vehicle-only treatment.

Changes in fetal weight near term (for example, embryonic day 18 for mice

and embryonic day 21 for rat); birth weight; and/or other measures of size at term or birth, such as length.

Animal study selection process

2,767 records identified

through database

searching

62 records identified through hand

searching (snowball searching)

2,049 records after

duplicates removed

2,049 titles and abstracts

screened

1,982 records excluded

67 full-text articles assessed

for eligibility

46 full-text articles excluded:•Duplicate data published in separate included study (23)

•No measurement or ineligible measurement of fetal growth (16)

•No PFOA exposure or ineligible exposure regimen (5)

•No original data (1)

•Preliminary abstract (unable to obtain data) (1)

21 studies (32 separate datasets)

included in qualitative synthesis

7 studies (8 separate datasets) included

in quantitative analysis (meta-analysis)

Summary of Study Characteristics

Species

Route of Exposures

Mouse Chicken Rat Fly Salmon Zebrafish

Gavage Injection into Egg Egg ImmersionFood Drinking Water Inhalation

Time point of Growth Measurement

At Birth Near Term Not Stated

Method of Growth Measurement

Weight Length Larval Volume

During larval development

Summary of Study Characteristics

Study data: Pup mammalian weight

Doses in figure decrease as y-axis increases**mg/kg BW/day unless otherwise specified

#Wolf study contributed two data sets—”a” exposed one group of animals from GD1-17 and “b” exposed a different group during a varied subset of days between GD1-17

Subset of studies for meta-analysis

Comparability across studies determinedbased on study characteristics:

• Animal model used:Mouse

• Developmental stage at measurement:Birth

• Outcome reported:Weight

• PFOA exposure:Oral Gavage (similar dose, frequency, timing, and duration)

Meta-analysis results: Decrease in birth weight with increase in PFOA exposure

Estimates a 0.023g decrease in birthweight

for every mg/kg/day increase in PFOA

exposure


Systematic search

Select Studies

Extract Data & Data

Analysis




Rate Strength of Evidence

Rate the Quality and Strength of the Evidence

High

Moderate

Low

Sufficient evidence of toxicity

Limited evidence of toxicity

Inadequate evidence of toxicity

Evidence of lack of toxicity

Animal risk of bias results

Hu 2010 [68]

Yahia 2010 [103]

Hines 2009 [260]

Fenton 2009 [264]

White 2009 [312]

Abbott 2007 [528]

White 2007 [566]

Wolf 2007 [571]

Lau 2006 [635]

Hinderliter 2005 [711]

Staples 1984 [1871]

Boberg 2008 [3061]

Onishchenko 2011 [3610]

White 2011 [3862]

York 2002 [5122]

Hagenaars 2011 [59]

Wang 2010 [86]

Pinkas 2010 [187]

O’Brien 2009 [236]

Jiang 2012 [3926]

Spachmo [3932]

Study [study ID]

Low risk

Probably low risk

Probably high risk

High risk

Mammalian population

Non-mammalian population

Factors for downgrading/upgrading evidence were derived directly from factors used in GRADE and Cochrane

High

Moderate

Low

Animal evidence• Level of study control• Randomization key factor in default of “high” in

GRADE• Studies find humans more susceptible to chemical

exposures than animals

Human evidence• Non-randomized, but variable quality. Cochrane

review finds similar results between RCTs and Observational studies

Rate the Quality of the Evidence

Quality of Mammalian Evidence

-1 Downgrade = Moderate Quality

DowngradeRisk of Bias Indirectness Inconsistency Imprecision Publication Bias

Final -1 0 0 0 0

Sequence generation

Allocation concealment

Blinding

Incomplete outcome data

Selective reporting

Other bias

Conflict of interest

0% 20% 40% 60% 80% 100%

Quality of Non-Mammalian Evidence

-2 Downgrade = Low Quality

DowngradeRisk of Bias Indirectness Inconsistency Imprecision Publication Bias

Final -1 -1 0 0 0

Sequence generation

Allocation concealment

Blinding

Incomplete outcome data

Selective reporting

Other bias

Conflict of interest

0% 20% 40% 60% 80% 100%

28

Evidence Stream

HumanNon-human mammalian

Starting rating Moderate High

Do

wn

grad

e Risk of Bias 0 -1

Indirectness 0 0

Inconsistency 0 0

Imprecision 0 0

Publication bias 0 0

Up

grad

e Large magnitude effect 0 N/A

Dose response 0 N/A

All possible confounding would confirm negative result

0 N/A

Grade 0 -1

Final rating Moderate Moderate

Summary of Quality of Evidence for PFOA

QUALITY OF EVIDENCE

Moderate

High

Low

STRENGTH OF EVIDENCE(LEVEL OF CERTAINTYREGARDING TOXICITY)


Limited evidence of toxicity

Inadequate evidence of toxicity

Evidence of lack of toxicity

CRITERIA:1. Quality of evidence:2. What is the direction of effect?3. What is the confidence in the

effect?4. Are there other compelling

attributes of the data that influence certainty?



CRITERIA:1. Quality of evidence: Moderate2. What is the direction of effect? Decrease in fetal growth

with PFOA exposure3. What is the confidence in the effect? Confidence based

on consistency on results and overlapping confidence intervals

4. Are there other compelling attributes of the data that influence certainty? None

Non-Human Mammalian Evidence = “Sufficient”

A positive relationship has been established through either multiple positive results or a single appropriate study in a single species. The available evidence includes results from one or more well-designed, well-conducted studies, and the conclusion is unlikely to be strongly affected by the results of future studies.


CRITERIA:1. Quality of evidence: Moderate2. What is the direction of effect? Decrease in fetal growth

with increasing PFOA exposure3. What is the confidence in the effect? A new study would

be unlikely to change the certainty in the direction of the effect

4. Are there other compelling attributes of the data that influence certainty? None

Human Evidence = “Sufficient”

A positive relationship is observed between exposure and outcome where chance, bias, and confounding can be ruled out with reasonable confidence. The available evidence includes results from one or more well-designed, well-conducted studies, and the conclusion is unlikely to be strongly affected by the results of future studies.

Sufficient Limited InadequateEvidence of

Lack of Toxicity

Sufficient Known to be Toxic to Human Reproduction

Limited Probably Toxic Possibly Toxic

Inadequate Possibly Toxic Not Classifiable

Evidence of Lack of Toxicity Not Classifiable

Probably NotToxic

Strength of Evidence in Non-Human Systems

Stre

ngt

h o

f Ev

ide

nce

in

H

um

an S

yste

ms

Conclusion: Review authors came to the final conclusion that “exposure to PFOA is ‘known to be toxic’ to human reproduction and development based on sufficient evidence of decreased fetal growth in both human and non-human mammalian species.”

Integrating the PFOA Streams of Evidence

• To assess the strength of the evidence for the reverse causality hypothesis using the Navigation Guide systematic review method [4] to answer the question: ‘Is there an association between fetal growth and maternal GFR in humans?’

Objective

A. B.

A. Changes in fetal growth may affect the concentration of measurable chemical due to changes in the maternal plasma volume and subsequent changes in maternal glomerular filtration rate.

Two potential hypotheses for the relationship between exogenous chemicals and fetal growth.

B. Increased exposure to exogenous chemicals may cause changes in fetal growth.

shaded green area represents the direction of effect (positive or negative in relation to zero (no effect)) which is consistent with the hypotheses for the change in normal pregnancy

Relationship between GFR and fetal growth – Human observation studies

shaded green area represents the direction

of effect (positive or negative in relation to

zero (no effect)) which is consistent with the hypotheses for the change in normal

pregnancy

Association between fetal growth and PVE in non-human studies

Evidence Stream Integration

Navigation Guide Compared to Narrative Reviews

Reference Study

question

Inclusion/

exclusion

criteria

Reprodu

cible

search

Risk of

Bias

Data

analysis

Summary

findings

table

Assess

quality of

evidence

Integrate

evidence

streams

Nar

rati

veR

evi

ew

s

Post et al 2012 YES - - - - YES - -Lindstrom et al 2011 YES - - - - - - -Stahl et al 2011 YES - - - - YES - -White et al 2011 YES - - - - YES - -Steenland et al 2010 YES - - - - - - -DeWitt et al 2009 YES - - - - - - -Olsen et al 2009 YES - - - - YES - -Jensen and Leffers 2008 - - - - - - - -Lau et al 2007 YES - - - - - - -Butenhoff et al 2004 YES YES - - Partial YES - -

Kennedy et al 2004 YES - - - - YES - -Lau et al 2004 YES - Partial - - - - -Hekster et al 2003 YES Partial Partial - - YES - -Kudo and Kawashima 2003 YES - - - - - - -

Navigation YES YES YES YES YES YES YES YES

October 2014

Conclusion

• We can do it

• Rigorous, systematic, transparent and doable

• Capacity to evolve with changes in evidence stream

Sir Austin Bradford Hill - incompleteness of science … “does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time” (Hill 1965).

Acknowledgements

PFOA Case Study Authors

AcknowledgementsNavigation Guide Work Group

Program on Reproductive Health and the Environment

Thank you

systematic review and meta-analysis for translating ... · systematic review and meta-analysis for...

Documents