steps 3 & 4: evaluating types of evidence for the truckee river case study
TRANSCRIPT
Steps 3 & 4: Evaluating types of evidence for the Truckee River case study
2
Define the Case
List Candidate Causes
Evaluate Data from Elsewhere
Identify Probable Cause
Detect or Suspect Biological Impairment
As Necessary: Acquire Data
and Iterate Process
Identify and Apportion Sources
Management Action: Eliminate or Control Sources, Monitor Results
Biological Condition Restored or Protected
Decision-maker and
Stakeholder Involvement
Stressor Identification
Step 3: Evaluate Data from the Case
Step 4: Evaluate Data from Elsewhere
3
Types of evidence using data from the case
• Spatial/temporal co-occurrence
• Evidence of exposure or biological mechanism
• Causal pathway
• Stressor-response relationships from the field
• Manipulation of exposure
• Laboratory tests of site media
• Temporal sequence
• Verified predictions
• Symptoms
Types of evidence using data from elsewhere
• Stressor-response relationships from other field studies
• Stressor-response relationships from laboratory studies
• Stressor-response relationships from ecological simulation models
• Mechanistically plausible cause
• Manipulation of exposure at other sites
• Analogous stressors
Use all available types of evidence to make an inferential assessment
italics indicates commonly available types of evidence
4
Basic analysis strategy
• Develop as many types of evidence, for as many candidate causes, as you can
– you won’t have all types of evidence, for all candidate causes– most effective when you can compare results across candidate
causes
• Work through one type of evidence, then set it aside – avoid cognitive overload
• Show your work– make your process transparent & reproducible – make use of appendices
5
Let’s begin by figuring out what types of evidence we have for the Truckee…
6
General vs. specific causation
• General – Does C cause E?– Does smoking cause lung cancer?
– Does increased water temperature reduce bull trout abundance in rivers?
• Specific – Did C cause E?– Did smoking cause lung cancer in
Ronald Fisher?
– Did increased water temperature reduce bull trout abundance in my stream?
7
Specific causation: using data from the case
8
Spatial/temporal co-occurrence
SUPPORTS
WEAKENS
Want paired measurements of proximate stressors & biological impairments, at locations where impairments are & are not observed.
9
Causal pathway
Want paired measurements of other steps in causal pathway & biological impairments, at locations where impairments are & are not observed.
10
Stressor-response relationships from the field
Want paired measurements of proximate stressors (or other steps in causal pathway) & biological impairments, at varying levels of exposure.
11
Other types of evidence using data from case
TYPE OF EVIDENCE SUPPORTING EVIDENCE
Manipulation of exposure Impairment improves after stressor is removed
Laboratory tests of site media
Exposure to site media in lab tests results in effects similar to impairment
Evidence of exposure or biological mechanism
Measurements of biota (e.g., biomarkers, tissue residues) show proposed mechanism of exposure has occurred
Verified predictions Predictions based on stressor’s mode of action are made & confirmed at site
Temporal sequence Exposure to stressor precedes impairment
Symptoms Only one stressor supports observed symptom
12
What types of evidence do we have, using data from the case?
13
General causation: using data from elsewhere?
Stressor-response relationships from other field studies
15
Stressor-response relationships from the lab
16
Other types of evidence using data from elsewhere
TYPE OF EVIDENCE SUPPORTING EVIDENCE
Stressor-response relationships from ecological simulation models
Stressor is at levels associated with impairment in mathematical models simulating ecological processes
Manipulation of exposure Impairment improves after stressor is removed at another site
Mechanistically plausible cause
Relationship between stressor & impairment is consistent with current scientific knowledge
Analogous stressors Stressor is structurally similar to other stressors known to cause impairment
Verified predictions Predictions based on stressor’s mode of action are made & confirmed at other sites
17
What types of evidence do we have, using data from elsewhere?
18
Now that we know what data we have, how do we analyze it?
19
Spatial co-occurrence
Do your impairment and your stressor co-occur in space?
To Do:1. Load relevant data file2. Merge files3. Make boxplots for each
candidate causeSelect ‘reference’ and impaired sites
4. Fill in worksheet
20
Causal Pathway
Does your data support the steps in the causal path between the stressor and the impairment?
To Do:1. Return to the conceptual
diagram2. Identify the steps in the causal
pathway3. Construct table to show
whether data supports the steps between the stressor and the impairment
4. Fill in worksheet
21
Verified Prediction - Traits
Do data support predictions based on stressor’s mode of action?
To Do:1. Load relevant data file2. Merge files3. Make boxplot
Select ‘reference’ and impaired sites
4. Fill in worksheet
22
Verified Prediction - PECBO
Do data support predictions based on stressor’s mode of action?
To Do:1. Load relevant data file2. Merge files3. Run PECBO4. Load PECBO results file into
CADStat5. Merge files6. Make boxplot
1. Sed2. STRMTEMP
7. Fill in worksheet
23
Stressor-response from elsewhere
Does impairment decrease as exposure to the stressor decreases (or increases as exposure increases)?
To Do:1. Listen and ask lots of
questions2. Fill in the worksheet
24
Randomized, controlled experiments
Key elements:• Replication: use of multiple
test units (e.g. tanks, sites)
• Controls: differ only by absence of the treatment
• Randomization: random assignment of test units to “control” or “treated” status
• Statistical analysis: estimate treatment effect (causal)
The scientific standard for establishing cause and effect
0
1
2
3
4
5
6
7
1 2 3 4
Toxin
Su
rviv
ors
ip
25
Observational studies
Key elements:• Replication: collect data
from multiple test units
• Controls: ?• Randomization: ?• Statistical analysis: identify
associations among variables of interest (non-causal)
Often the only option for large-scale field studies
None
None
26
Trade offs: control vs. realism, scale
Lab Experiment
Field Experiment
Observational Study
control
realism,scale
27
Biomonitoring = Observational
Issues for causal analysis:
• Estimates of stressor effects are confounded by covarying factors
• Analyst can’t randomly assign treatments (stressors) to sites
* Reference sites are not experimental controls
28
Analogous to clinical trials
Does smoking cause lung cancer?
• Estimates of stressor effects are confounded by covarying factors
• Analyst can’t randomly assign treatments (stressors) to subjects
* Non-smokers without lung cancer are not experimental controls
29
Example using western EMAP*
Using propensity scores to infer cause-effect relationships in observational data
– Analysis and slides by Lester Yuan (USEPA), Amina Pollard (USEPA), and Daren Carlisle (USGS)
– Original presentation given at North American Benthological Society conference, May 2008
*EPA Environmental Monitoring and Assessment Program (EMAP)
30
EMAP-West Study Area
Measurements Collected:
•Macroinvertebrates
•Substrate composition
(SED)
•Stream temperature
(STRMTEMP)
•N = 838
Data collected by the EPA Environmental Monitoring and Assessment Program (EMAP)
31
Total N vs. total taxon richness
Data from EMAP Western Pilot
SLOPE = -16.5
32
Total N covaries with many other factors
33
Multiple linear regression
Include covariates in the regression model to control for their effect.
SLOPE = -9.4
Regression model includes: %agriculture, %urban, grazing intensity, %sands/fines, stream temperature, and log conductivity.
SLOPE = -16.5
Correlation of Total Richness and Total N (ug/L)
34
Potential issues with multiple regression
• Must assume that linear relationships are appropriate for all covariates.
• Regression model may extrapolate.
• Inclusion of certain variables may “mask” true effect:– e.g., part of the effect of agriculture may be
attributed to total N
35
Alternate approach: Stratify dataset
r = -0.01 r = 0.15 r = 0.27
r = 0.64
36
Model richness vs. total N within strata
How do we simultaneously stratify on many different covariates?
SLOPE = -10.7 SLOPE = -12.3 SLOPE = -9.7
37
Propensity Score Matching
• Method developed in epidemiology to retroactively control for confounding effects in observational studies
• Sometimes called a quasi-experiment• Intuitively:
1. Model the magnitude of treatment (e.g. nutrient concentration) as a function of the covariates. The predicted magnitude of treatment at each site is its propensity score.
2. Stratify the total set of observations by the propensity scores (i.e., group sites with similar scores). Six strata are typically used.
3. Within each stratum, sites having different treatment levels (e.g. high vs. low nutrients) may be considered to have been “randomly assigned” to those treatment levels, because covariates have effectively been controlled by propensity score matching of “treated” and “control” sites.
38
Propensity Score Model
Total N = f(percent agriculture, percent urban, grazing intensity, percent sand/fines, stream temperature, log conductivity, elevation, log catchment area, canopy cover, sampling day)
39
Define 6 strata based on propensity scores
40
Covariate values within strata
Grazing intensity
Percent agriculture
41
Stratification by propensity score controls covariance of all modeled variables
Original r Min r Max r
Percent agriculture 0.60 0.06 0.24
Percent urban 0.29 -0.12 0.40
Grazing intensity 0.64 -0.21 0.18
Percent sand/fines 0.64 -0.16 0.13
Stream temperature 0.48 -0.13 0.22
log conductivity 0.68 -0.12 0.16
Elevation -0.26 -0.38 0.20
log catchment area 0.48 -0.26 0.21
Canopy cover 0.45 -0.13 0.09
Sampling day -0.17 -0.21 0.10
After stratification
42
Total N vs. total taxon richness
Data from EMAP Western Pilot
SLOPE = -16.5
43
Total N vs. total richness: Stratified
SLOPE = -3.3 (n.s.)
SLOPE = -10.0***
SLOPE = -7.1*SLOPE = -7.1*
SLOPE = -10.5*** SLOPE = -8.1***