review and recommendation of methods for …frey/reports/mokhtari_frey_2005b.pdfi preface this is...
TRANSCRIPT
Review and Recommendation of Methods for Sensitivity and Uncertainty Analysis for the Stochastic Human
Exposure and Dose Simulation (SHEDS) Models
Volume 2: Evaluation and Recommendation of Methodology for Conducting
Sensitivity Analysis in Probabilistic Models
2004-206-01
Prepared by:
Amirhossein Mokhtari H. Christopher Frey
Department of Civil, Construction, and Environmental Engineering North Carolina State University
Raleigh, NC
Prepared for:
Alion Science and Technology 1000 Park Forty Plaza
Durham, NC
June 30, 2005
i
Preface
This is one of a two volume series of reports on the topic of review and recommendation
of methods for sensitivity and uncertainty analysis for the Stochastic Human Exposure and Dose
Simulation (SHEDS) models.
The first volume provides a comprehensive review of methods for sensitivity and
uncertainty analysis, with a focus on methods that are relevant to probabilistic models based
upon Monte Carlo simulation or similar techniques for propagation of distributions for variability
and uncertainty in model inputs in order to estimate variability and uncertainty in model outputs.
The methods included in the review are those that are considered to be “available,” which is
interpreted to be methods of practical significance as opposed to all possible methods that have
been proposed but not tested in practice. For each method, there is a description of the method,
followed by a discussion of the advantages and disadvantages of the method. A framework for
selection of sensitivity analysis methods is presented, leading to recommendations for a more
narrow set of such methods that merit more detailed evaluation.
The second volume proposes and applies a methodology for evaluation of the selected
sensitivity analysis methods based upon application of each method to a modeling testbed. The
testbed is a simplified version of a typical SHEDS model. A case study scenario was defined
that includes multiple time scales (e.g., daily, monthly). Seven sensitivity analysis methods were
applied to the testbed, including Pearson correlation, Spearman correlation, sample regression,
rank regression, Analysis of Variance (ANOVA), Fourier Amplitude Sensitivity Test (FAST),
and Sobol’s method. The sensitivity analysis results obtained from these seven methods were
compared. On the basis of these quantitative results, recommendations were made for methods
that offer promise for application to SHEDS models. The statistically-based methods are often
ii
readily available features of commonly available software packages. However, FAST and
Sobol’s method are less readily available. Therefore, algorithms are presented for these two
methods. Furthermore, recommendations are made for additional research and development of
sensitivity analysis methods for application to the SHEDS models.
iii
TABLE OF CONTENTS
1. INTRODUCTION..............................................................................................................1
2. OVERVIEW OF THE STOCHASTIC HUMAN EXPOSURE AND DOSE SIMULATION (SHEDS) MODELS ................................................................................5 2.1 Overview of the SHEDS-Pesticides Model .............................................................5
2.2 Main Characteristics of the SHEDS-Pesticide Model .............................................8
2.3 Simplified Version of the SHEDS-Pesticides Model ..............................................9
3. OVERVIEW OF UNCERTAINTY AND SENSITIVITY ANALYSIS METHODS .......................................................................................................................13 3.1 Methods for Propagation and Quantification of Variability and
Uncertainty.............................................................................................................13 3.1.1 Analytical Propagation Techniques...........................................................13 3.1.2 Approximation Methods.............................................................................14 3.1.3 Numerical Propagation Techniques ..........................................................14
3.2 Sensitivity Analysis ...............................................................................................16 3.2.1 Correlation Analysis ..................................................................................16 3.2.2 Linear Regression Analysis........................................................................17 3.2.3 Analysis of Variance ..................................................................................17 3.2.4 Sobol’s Method ..........................................................................................18 3.2.5 Fourier Amplitude Sensitivity Test (FAST)................................................20
4. DEFINING CASE SCENARIOS FOR SENSITIVITY ANALYSIS ..........................23 4.1 Probabilistic Dimensions .......................................................................................24
4.2 Susceptible Subpopulation.....................................................................................25
4.3 Identification of Pathway of Interest......................................................................26
4.4 Time Scales of the Model Simulation....................................................................26
5. RESULTS OF SENSITIVITY ANALYSIS FOR CASE STUDIES ...........................31 5.1 Model Application .................................................................................................31
5.2 Correlation Analysis ..............................................................................................33 5.2.1 Correlation Coefficients Results ................................................................34 5.2.2 Rankings for Selected Inputs......................................................................47 5.2.3 Comparison of Mean Ranks.......................................................................52 5.2.4 Summary of Results Based on Correlation Coefficients ............................55
5.3 Regression Analysis...............................................................................................58 5.3.1 Coefficient of Determination......................................................................58 5.3.2 Regression Coefficients..............................................................................61 5.3.3 Rankings for Selected Inputs......................................................................67
iv
5.2.4 Comparison of Mean Ranks.......................................................................71 5.2.5 Summary of Results Based on Regression-Based Methods .......................74
5.4 Analysis of Variance..............................................................................................76 5.4.1 F Values .....................................................................................................77 5.4.2 Rankings for Selected Inputs......................................................................78 5.4.3 Comparison of Mean Ranks.......................................................................80 5.4.5 Summary of Results Based on ANOVA ......................................................82
5.5 Sobol’s Method......................................................................................................83 5.5.1 General Insight Regarding Contribution of Inputs to the Output
Variance.....................................................................................................83 5.5.2 Sensitivity Indices.......................................................................................89 5.5.3 Ranking for Selected Inputs .......................................................................93 5.5.4 Comparison of Mean Ranks.......................................................................95 5.5.5 Summary of Results Based on Sobol’s Method..........................................98
5.6 Fourier Amplitude Sensitivity Test........................................................................99 5.6.1 General Insight Regarding Contribution of Inputs to the Output
Variance.....................................................................................................99 5.6.2 Sensitivity Indices.....................................................................................101 5.6.3 Ranking for Selected Inputs .....................................................................105 5.6.4 Comparison of Mean Ranks.....................................................................106 5.6.5 Summary of Results Based on FAST........................................................109
5.7 Comparison of Results for Selected Sensitivity Analysis Methods.....................110
6. CONCLUSIONS AND RECOMMENDATIONS.......................................................115 6.1 Main Characteristics of the SHEDS Models Relevant to the Process of
Choosing Appropriate Sensitivity Analysis Methods..........................................115
6.2 Available Sensitivity Analysis Methods..............................................................115
6.3 Sensitivity Analysis Methods for Application to the SHEDS Models ................116
6.4 Recommendations................................................................................................125
REFERENCES...........................................................................................................................131
APPENDIX A .............................................................................................................................139
APPENDIX B .............................................................................................................................149
v
LIST OF FIGURES
Figure 2-1. Schematic Diagram of the SHEDS-Pesticides Simulation Process. ...................7 Figure 2-2. Schematic Diagram for the Simplified SHEDS-Pesticides Model. ....................9 Figure 4-1. Components of a Scenario for Sensitivity Analysis of the Simplified
SHEDS-Pesticides Model. ............................................................................24 Figure 4-2. Probability Distribution of Different Average Exposure Pathways in 30
Days of Model Simulation. ...........................................................................27 Figure 4-3. Example of Probabilistic Results for Variation in the Total Exposure
for Selected Random Individuals: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure. .............................................................................29
Figure 5-1. Schematic Algorithm for Application of Selected Sensitivity Analysis Methods to the Simplified SHEDS-Pesticides Model. .................................33
Figure 5-2. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and Spearman Correlation Coefficients for Re sidue Decay Rate as an Input with Monthly Sampling Frequency for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure. ............35
Figure 5-3. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and Spearman Correlation Coefficients for Probability of Washing Body as an Input with Monthly Sampling Frequency for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure. ........41
Figure 5-4. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and Spearman Correlation Coefficients for Fraction of Chemicals Available for Transfer as an Input with a Daily Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure..........................................................................44
Figure 5-5. Variation in Rank of Residue Decay Rate as an Input with Monthly Sampling Strategy with Respect to Time Based on: (a) Pearson; and (b) Spearman Correlation Coefficients. ........................................................48
Figure 5-6. Variation in Rank of Fraction of Chemical Available for Transfer as an Input with Daily Sampling Strategy with Respect to Time Based on: (a) Pearson; and (b) Spearman Correlation Coefficients. .............................50
Figure 5-7. Comparison of Mean Ranks and Range of Ranks Representing Minimum and Maximum Ranks of Inputs in 30 Days of Model Simulation for the Three Temporal Scenarios Based on: (a) Pearson Correlation Coefficients; and (b) Spearman Correlation Coefficients. ........53
Figure 5-8. Variation in Coefficient of Determination, R2, with Respect to Time based on Sample and Rank Regression Analyses for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Exposure. ................................60
Figure 5-9. Absolute Values and Corresponding 95% Confidence Intervals of Sample and Rank Regression Coefficients for Residue Decay Rate as
vi
an Input with a Monthly Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure. ......................63
Figure 5-10. Absolute Values and Corresponding 95% Confidence Intervals of Sample and Rank Regression Coefficients for Probability of Washing Body as an Input with a Monthly Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure. ........66
Figure 5-11. Absolute Values and Corresponding 95% Confidence Intervals of Sample and Rank Regression Coefficients for Fraction of Chemicals Available for Transfer as an Input with a Daily Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure. ......................................................................................................68
Figure 5-12. Variation in Rank of Residue Decay Rate as an Input with Monthly Sampling Strategy with Respect to Time Based on: (a) Sample Regression Analysis; and (b) Rank Regression Analysis. ............................69
Figure 5-13. Variation in Rank of Fraction of Chemical Available for Transfer as an Input with a Daily Sampling Strategy with Respect to Time Based on: (a) Sample Regression Analysis; and (b) Rank Regression Analysis.........................................................................................................70
Figure 5-14. Comparison of Mean Ranks and Range of Ranks Representing Minimum and Maximum Ranks of Inputs in 30 Days of Model Simulation for the Three Temporal Scenarios Based on: (a) Sample Regression Analysis; and (b) Rank Regression Analysis. ............................72
Figure 5-15. Variation of F Values with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer. ................................................................78
Figure 5-16. Variation of Ranks with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer. ................................................................79
Figure 5-17. Comparison of Mean Ranks and Range of Ranks of Inputs in 30 Days for the Three Temporal Scenarios Based on ANOVA. ................................81
Figure 5-18. Variation in Average Sum of Main Effects of Inputs in 500 Bootstrap Simulations and Corresponding Uncertainty Ranges with Respect to Time for the Three Temporal Scenarios. (Intervals shown represent plus or minus one standard error of the mean value)....................................85
Figure 5-19. Contribution Selected Inputs to the Output Variance in Scenario I for Selected Days: (a) Day 1; (b) Day 15; and (c) Day 30. ...............................87
Figure 5-20. Average Main and Total Effects and Corresponding Uncertainty Ranges Based on 500 Bootstrap Simulations for Residue Decay Rate for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure. ......................................................................................................91
Figure 5-21. Average Main and Total Effects and Corresponding Uncertainty Ranges Based on 500 Bootstrap Simulations for Residue Decay Rate
vii
for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure. ......................................................................................................92
. 92 Figure 5-22. Variation of Ranks with Respect to Time for the Three Temporal
Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer Based on the Sobol’s Method.................94
Figure 5-23. Comparison of Mean Ranks and Range of Ranks Based on the Sobol’s Method for Inputs in 30 Days for the Three Temporal Scenarios Based on: (a) Main Effects; and (b) Total Effects. ......................................97
Figure 5-24. Variation in Sum of Main Effects of Inputs Based on FAST with Respect to Time for: (a) Daily; (b) Incremental; and (c) Cumulative Total Exposure Scenarios. ..........................................................................100
Figure 5-25. Main and Total Effects for Residue Decay Rate Based on FAST for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure. .................103
Figure 5-26. Main and Total Effects for Fraction of Chemicals Available for Transfer Based on FAST for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure. ...........................................................................104
Figure 5-27. Variation of Ranks with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer Based on FAST. ...................................106
Figure 5-28. Comparison of Mean Ranks and Range of Ranks Based on the FAST for Inputs in 30 Days for the Three Temporal Scenarios Based on: (a) Main Effects; and (b) Total Effects. ......................................................108
Figure 5-29. Comparison of the Mean Ranks and Corresponding Range of Ranks in 30 Days of Model Simulation Based on Pearson Correlation Analysis (PCA), Spearman Correlation Analysis (SCA), Sample Regression Analysis (SRA), Rank Regression Analysis (RRA), Analysis of Variance (ANOVA), Sobol’s Method, and FAST for: (a) Inputs with Monthly Sampling Frequencies; and (b) Inputs with Daily Sampling Frequencies in Scenario II ................................................112
Figure A.1. Schematic Diagram of the Simplified SHEDS Model. ..................................140 Figure B-1: Matlab Code for Assigning Frequencies to Daily Inputs. ..............................155 Figure B-2: Matlab Code for Selecting a Search Curve for Inputs with Uniform,
Loguniform, and Normal Distributions. .....................................................156 Figure B-3: Matlab Code for Selecting a Search Curve for Inputs with Lognormal
Distribution. ................................................................................................157 Figure B-4: Matlab Code for Estimating the Output Variance..........................................158 Figure B-5: Matlab Code for Estimating the Fourier Coefficients. ...................................159 Figure B-6: Matlab Code for Assigning New Frequencies to Daily Inputs Used in
Estimation of Total Sensitivity Indices.......................................................160 Figure B-7. Flow Diagram for Application of the Sobol’s Method. .................................166 Figure B-8. Forming Mj and Nj Matrices for Input xj with Monthly Sampling
Frequency....................................................................................................167
viii
Figure B-9. Forming Mj and Nj Matrices for Input xj with Daily Sampling Frequency....................................................................................................168
Figure B-10. Matlab Code for Forming M and N Matrices for Inputs with Monthly and Daily Sampling Frequencies. ...............................................................169
ix
LIST OF TABLES
Table 2-1. Examples for Input Assumptions for the Simplified SHEDS-Pesticides Model ........11 Table 4-1. Example of Probabilistic Results for Daily, Incremental Change in Daily, and
Cumulative Total Exposure for the Random Individual ‘a’ shown in Figure 4-3 ..........................................................................................................................30
Table 6-1. Comparison of Various Sensitivity Analysis Methods .............................................117 Table A.1. Input Assumptions for the Simplified SHEDS model ..............................................146
1
1. INTRODUCTION
The purpose of this project is to identify, evaluate, and recommend methods for
sensitivity analysis applicable to the Stochastic Human Exposure and Dose Simulation (SHEDS)
risk assessment models. The EPA SHEDS models are aggregate, probabilistic and physically-
based human exposure models that simulate variability and uncertainty in cumulative human
exposure and dose (EPA, 2000; Price et al., 2003). Understanding key sources of variability can
guide the identification of significant subpopulations that merit more focused study and in
developing approaches for risk management. In contrast, knowing the key sources of uncertainty
can aid in determining whether additional research or alternative measurement techniques are
needed to reduce uncertainty (Cullen and Frey, 1999).
Volume I of this series of report provides an overview of motivations for conducting
sensitivity and uncertainty analyses. Sensitivity analysis is recognized as an essential component
of analyses for complex models (Helton, 1997). Sensitivity analysis highlights the inputs that
have the greatest influence on the results of a model; therefore, it provides useful insights for
model builders and users (McCarthy et al., 1995). Sensitivity analysis provides insight regarding
which model input contributes the most to uncertainty, variability, or both, for a particular model
output. Insights from sensitivity analysis can be used for: (1) identification of key sources of
uncertainty; (2) identification of key controllable sources of variability; and (3) model
refinement, verification, and validation (Mokhtari and Frey, 2005).
Sensitivity analyses of risk models are used to identify inputs that matter the most with
respect to exposure or risk and aid in developing priorities for risk mitigation and management
(Baker et al., 1999; Jones, 2000). Sensitivity analysis can identify important uncertainties.
Additional data collection or research can be prioritized based on the key uncertainties in order
2
to reduce the model output uncertainty (Cullen and Frey, 1999). Knowledge of key controllable
sources of variability is useful in identifying control measures to reduce exposure or risk.
Sensitivity analysis has been used for verification and validation purposes during the
process of model development and refinement (e.g., Kleijnen, 1995; Kleijnen and Sargent, 2000;
Fraedrich and Goldberg, 2000). Sensitivity analysis can be used for verification by assessing
whether the model output responds appropriately to a change in model inputs. Sensitivity
analysis can validate a model by determining the degree to which a model is an accurate
representation of the real world. Sensitivity analysis also can be used to evaluate the robustness
of model results (e.g., Philips et al., 2000; Ward and Carpenter, 1996; Limat et al., 2000;
Manheim, 1998; Saltelli et al., 2000).
There are many sensitivity analysis methods applied in various scientific fields, including
engineering, economics, physics, social sciences, medical decision making, and others (e.g.,
Baniotopoulos, 1991; Cheng, 1991; Merz et al., 1992; Helton and Breeding, 1993; Beck et al.,
1997; Agro et al., 1997; Kewley et al., 2000; Oh and Yang, 2000). Given the myriad of
sensitivity analysis methods, there is a need for insight regarding which methods to choose and
regarding how to apply preferred methods. In Volume 1, approximately a dozen available
sensitivity analysis methods were identified and reviewed, including their advantages and
disadvantages. Based upon the review, a selected set of seven sensitivity analysis methods was
selected for more detailed and quantitative evaluation. These methods include Pearson and
Spearman correlation , sample and rank regression, analysis of variance (ANOVA), Fourier
amplitude sensitivity test (FAST), and Sobol’s method. In this volume, these selected methods
are applied to a testbed in order to facilitate comparisons based upon quantitative results. The
testbed is a simplified version of a typical SHEDS model.
3
A key objective of the sensitivity analysis of the SHEDS models was to quantify the
contribution of individual inputs to the variance in the selected model output. Thus, sensitivity
analysis methods were selected for further evaluation based on their capability in proving
insights with respect to the key objective. Among the selected techniques, FAST and Sobol’s
method are variance-based methods and provide quantitative measures for contributions of
inputs to the output variance. Although selected sampling-based methods do not provide
measures regarding the contributions of inputs to the output variance, comparison of the results
based on these methods and the two variance-based techniques can provide insights regarding
trade-off between application of commonly used sensitivity analysis methods and more
sophisticated techniques.
The key questions that are addressed in this report include the following:
(1) What are the main characteristics of the SHEDS models that are relevant to the
process of choosing appropriate sensitivity analysis methods?
(2) Which sensitivity analysis methods described in Volume 1 are most promising
and merit more detailed evaluation?
(3) What are the recommended sensitivity analysis methods for application to the
SHEDS models?
(4) What are the key recommendations for further research based on the insights from
the analyses?
Chapter 2 briefly explains the SHEDS models including their key characteristics. Furthermore, a
simplified version of the SHEDS-Pesticides model developed for evaluation of sensitivity
analysis methods is discussed. Chapter 3 provides a brief overview of uncertainty and sensitivity
analysis methods that are of practical significance. Chapter 4 defines a set of case scenarios for
4
evaluation of selected sensitivity analysis methods. Key attributes of each scenario including
probabilistic features, susceptible subpopulations, exposure pathway of interest, and time scale of
the model simulation are explained. Chapter 5 provides results from application of selected
sensitivity analysis methods to the case scenarios defined in Chapter 4. Chapter 6 summarizes
the conclusions and recommendations based on the analyses provided in Chapter 5 and discusses
key recommendations for further research. Appendix A provides the key equations of the
simplified SHEDS-Pesticides model along with the probability distributions of model inputs.
Appendix B provides algorithms for application of two of the sensitivity analysis methods
evaluated in this report including Fourier Amplitude Sensitivity Test (FAST) and the Sobol’s
method.
5
2. OVERVIEW OF THE STOCHASTIC HUMAN EXPOSURE AND DOSE SIMULATION (SHEDS) MODELS
SHEDS is a family of models that are developed to estimate multimedia and multi-
pathway pollutant exposures of general as well as at-risk populations. The SHEDS models are
being designed to predict and diagnose complex relationships between pollutant sources and
dose received by different subpopulation (e.g., children and the elderly) (Price et al., 2003). The
SHEDS models provide estimates of variability and uncertainty in the predicted exposure
distributions and characterize factors influencing high-end exposures. These models address both
aggregate (all sources, routes, and pathways for a single chemical) and cumulative (aggregate for
multiple chemicals) exposures. The SHEDS models include SHEDS-Pesticides, SHEDS-Wood,
SHEDS-Air Toxics and SHEDS-PM (EPA, 2000). The discussion presented here is mainly based
upon SHEDS-Pesticides, because this model has typical characteristics of the SHEDS series of
models.
Sections 2.1 and 2.2 provide an overview of and briefly explain the main characteristics
of the SHEDS-Pesticides model, respectively. For the purpose of evaluation of sensitivity
analysis methods, a simplified version of the SHEDS-Pesticides model is developed. The
simplified model has the key advantage of shorter simulation time compared to the original
model. The key assumptions of the simplified SHEDS-Pesticides model are briefly explained in
Section 2.3.
2.1 Overview of the SHEDS-Pesticides Model
The EPA SHEDS-Pesticides model is an aggregate, probabilistic and physically-based
human exposure model which simulates variability and uncertainty in cumulative human
exposure and dose to pesticides. The key purposes of the model are: (1) to improve the risk
assessment process by predicting both inter-individual variability and uncertainties associated
6
with population exposure and dose distributions; (2) to improve the risk management process by
identifying critical exposure routes and pathways; and (3) to provide a framework for identifying
and prioritizing measurement needs and for formulating the most appropriate hypotheses and
designs for exposure studies. The schematic diagram of the simulation process for the SHEDS-
Pesticides model is given in Figure 2-1.
The SHEDS-Pesticides model predicts, for user-specified cohorts, exposures and doses
incurred via eating contaminated foods or drinking water, inhaling contaminated air, touching
contaminated surface residues, and ingesting residues from hand-to-mouth activities. The model
combines information on pesticide usage, human activity data, environmental residues and
concentrations, exposure and dose factors. The SHEDS-Pesticides model is limited to pesticides
and other dislodgeable compounds present on surfaces in residential environments (Price et al.,
2003).
For each individual, the SHEDS-Pesticides model can construct daily exposure and dose
time profiles for the inhalation, dietary and non-dietary ingestion, and dermal contact exposure
routes. The dose profiles are then aggregated across routes to construct an individual 1-year
profile. Exposure and dose metrics of interest (e.g., peak, time-averaged, time-integrated) are
extracted from the individual’s profiles, and the process is repeated thousands of times to obtain
population distributions. This approach allows identification of the relative importance of routes,
pathways, and model inputs. Two-stage Monte-Carlo sampling is applied to allow explicit
characterization of both variability and uncertainty in model inputs and outputs.
The outputs from the SHEDS-Pesticides model include: (1) the time profile of exposure
and dose metrics for a specified post-application time period for an individual; (2) exposure and
dose contribution by routes and pathways; (3) cumulative density function (CDF) and box-
7
Figure 2-1. Schematic Diagram of the SHEDS-Pesticides Simulation Process.
Characterize Variability and Uncertainty in Population
Read Scenario Information
Sample an Individual
Collect Pesticides Usage Information
Simulate Individual 1-year Route-Specific Daily Exposure Rate
Simulate Dose Over All Routes
Two-Stage Monte Carlo Simulation
Variability Run
Uncertainty Run
8
whisker graphs for aggregate population estimates and uncertainty of percentiles.
2.2 Main Characteristics of the SHEDS-Pesticide Model
The main characteristics of the SHEDS-Pesticides model include: (1) non-linearity and
interaction between inputs; (2) saturation points; (3) different input types (e.g., continuous versus
categorical); and (4) aggregation and carry-over effects.
Non-linearity is a relationship between two variables in which the change in one variable
is not simply proportional to the change in the other variable. An example in the SHEDS-
Pesticides model is an exponential decay term in the airborne concentration of the pollutant
(Equation A.3 in Appendix A).
Interaction is a case in which the effect of an input depends on the value of another input.
Interaction terms can be introduced in a model in multiplicative forms.
A saturation point is an input value above which the model output does not respond to
changes in the input. For example, there is an upper limit for possible dermal exposure via body
and hand in the SHEDS-Pesticides model (Equation A.8 in Appendix A).
Inputs may be qualitative (categorical) (e.g., gender) or quantitative (e.g., pesticide
concentration). Quantitative inputs can be continuous (e.g., body weight) or discrete (e.g.,
number of application of a pesticide). Some quantitative inputs may be described by empirical
distributions, while other inputs may be represented by parametric distributions.
Aggregation refers to situations in which multiple numerical values are combined into
one numerical value, such as sum or mean value. An example is the total exposure (ETotal) in the
SHEDS-Pesticides model that is an aggregate of exposure via inhalation (EInhalation), dermal
(EDermal), and ingestion (EIngestion) pathways.
9
Figure 2-2. Schematic Diagram for the Simplified SHEDS-Pesticides Model.
The carry-over effect is a situation in which the exposure for an individual in each day is
a function of exposure in the same day and the prior day(s). For example, dermal exposure via
hand is corrected for the carry-over effect (Equation A.11 in Appendix A).
2.3 Simplified Version of the SHEDS-Pesticides Model
A simplified version of the SHEDS-Pesticides model is developed for evaluation of
sensitivity analysis methods. Figure 2-2 shows a schematic diagram for the simplified SHEDS-
Pesticides model. The final exposure is an aggregate of possible exposures via three pathways
including inhalation, dermal, and ingestion. Key equations of the simplified model along with
probability distributions of model inputs are given in Appendix A.
Two key points are considered when developing the simplified model: (1) the simplified
model should have the key characteristics of the SHEDS-Pesticides model; and (2) the
simulation time for the simplified model should be realistically practical. The results from the
simplified model are not real exposures. However, these values can be used for evaluation of
10
selected sensitivity analysis methods. In order to address the key points, the following
assumptions are made:
• Because the main objective of this evaluation is to identify which model inputs are major
contributors to the variance of model outputs, the effect of specific human activity
patterns and application patterns of pesticides on model outputs are not considered.
• One-stage Monte Carlo simulation is considered for the probabilistic framework of the
simplified model with a focus on variability in inputs.
• Exposure duration is set for one month in order to reduce the simulation time.
• There is only one application of pesticides during exposure period. The application
occurs at the first day of exposure duration.
• Daily exposures from multiple exposure pathways are calculated based upon randomly
generated exposure times corresponding to pathways, rather than based upon the human
time-activity pattern database used by the SHEDS-Pesticides model. This will help
reduce the complexity of simulation process and the simulation time.
Table 2-1 provides a summary of example inputs to the simplified SHEDS-Pesticides model
including their probability distribution and sampling frequencies. A complete list of inputs is
provided in Appendix A.
Typically, inputs have two different sampling strategies within the time scale of the
model simulation (e.g., one month): (1) monthly; and (2) daily. Inputs with a monthly sampling
strategy are sampled only once for each individual within each month, while in contrast, inputs
with a daily sampling strategy are sampled every day for each individual within the time scale of
the model simulation.
11
Table 2-1. Examples for Input Assumptions for the Simplified SHEDS-Pesticides Model
Inputs Distribution (a) Unit Comment Sampling Frequency
DR Triangular (0.1, 0.2, 0.4) [-] Fraction of residue that dissipates daily
monthly
PWB Triangular (0.3, 1.0, 1.0) [-] Probability of washing body monthly
FTR Uniform (0.0, 0.08) [-] Fraction of chemical available for transfer daily
a For triangular distributions, minimum, most likely, and maximum values are given For uniform distributions, minimum and maximum values are given
Examples of inputs with a monthly sampling strategy include fraction of residue that
dissipate daily (DR) and probability of washing body (PWB). DR represents a decay rate for
chemical residue on a daily basis. PWB represents a probability that an individual washes his/her
body in each day, and hence, increasing the chance of removing the chemicals from the body
surface. The fraction of chemicals available for transfer (FTR) is an input with a daily sampling
strategy. These inputs are later used in Chapter 5 when examples are provided to illustrate the
results of each sensitivity analysis method. Typically, these inputs are selected among the list of
top sensitive inputs in the model by different sensitivity analysis methods.
13
3. OVERVIEW OF UNCERTAINTY AND SENSITIVITY ANALYSIS METHODS
The objective of this chapter is to briefly review typical methods for: (1) propagation of
probability distributions of inputs through a model (Section 3.1); and (2) sensitivity analysis of a
model (Section 3.2). More detailed discussion methods for propagating probability distributions
of inputs through a model and of sensitivity analysis methods is given in Volume 1. This chapter
provides a brief summary of these methods for the convenience of the reader. Sensitivity analysis
methods introduced in this chapter are further evaluated through a set of case studies in Chapter
5.
3.1 Methods for Propagation and Quantification of Variability and Uncertainty
In order to quantify variability and uncertainty, the distribution of model inputs should be
propagated through the model to obtain distributions on model outputs. Propagation techniques
may be analytical, approximation, or numerical.
3.1.1 Analytical Propagation Techniques
For simple models in which the output is a function of linear combination of model
inputs with no dependency, the propagation of uncertainty through the model is straightforward.
For example, considering the Central Limit Theorem, the mean and variance of the output are
equal to the sum of the means and variance of the inputs (Wilson and Crouch, 1981; DeGroot,
1986). The same approach can be used for models with outputs as functions of multiplicative
independent input using a logarithmic transformation. Although analytical propagation is
conceptually straightforward and easy to implement, its applicability is limited to models with
linear summation (or product) of independent inputs.
14
3.1.2 Approximation Methods
Typical approximation methods for propagation of uncertainty in a model are generation
of system moments and first order methods (Hahn and Shapiro, 1967). These methods typically
use Taylor series expansions to propagate the moments of input distributions (e.g., mean,
variance, skewness, and kurtosis) through a model to quantify the equivalent moments of the
output values. The distribution of the output values can be approximated with propagated
moments using parametric distributions. Approximation methods typically have three major
limitations (Cullen and Frey, 1999). These methods require differentiable functional form for the
model, and hence, they are not applicable to models with discrete or discontinuous behaviors.
Approximation methods are also computationally intensive. Moreover, information regarding the
tails of the input distributions cannot be propagated with approximation methods.
3.1.3 Numerical Propagation Techniques
For most real-world applications involving complex and nonlinear model, analytical
techniques and often approximation methods are not favorable. The most common techniques for
numerical propagation of uncertainty and variability are sampling based methods. Sampling
based methods do not require access to the model equations. These methods involve running a
set of model simulations at a set of sampled points from probability distributions of inputs and
establishing a relationship between inputs and outputs using the model results. Some of the
commonly used sampling based uncertainty analysis methods are: (1) Monte Carlo Methods; (2)
Latin Hypercube Sampling methods; (3) Fourier Amplitude Sensitivity Test (FAST); and (3)
reliability based methods.
Monte Carlo Simulation. Monte Carlo simulation (Rubinstein, 1981; Doll and
Freeman, 1986; Kalos and Whitlock, 1986; Sobol, 1994; Fishman, 1996) is widely used for
15
propagating probability distributions of inputs through a model. Monte Carlo simulation involves
random sampling from distribution of inputs and successive model simulation to estimate the
distribution for the model output. Monte Carlo simulation can be used to solve problems with
physical probabilistic structures, such as uncertainty propagation in models or solutions of
stochastic equations, or can be used to solve non-probabilistic problems, such as finding the area
under a curve (Gardiner, 1983; Devroye, 1986). The applicability of Monte Carlo simulation is
sometimes limited to simple models because this method typically requires large number of
samples. For computationally intensive models, the time and resources required for Monte Carlo
simulation can be prohibitive.
Latin Hypercube Sampling. LHS method (Iman and Conover, 1980; Stein, 1987; Loh,
1996) is a variant of the standard Monte Carlo simulation. In LHS the probability distribution for
the random variable of interest is first divided into ranges of equal probability, and one sample is
taken from each equal probability range. However, the order of the samples is typically random
over the course of the simulation, and the pairing of samples between two or more random inputs
is usually treated as independent (Cullen and Frey, 1999). LHS method typically provides a
better coverage of the entire range of the probability distribution of an input compared to Monte
Carlo simulation.
Fourier Amplitude Sensitivity Test (FAST). FAST has been developed for uncertainty
and sensitivity analysis (Cukier et al., 1973; 1975, 1978; Schaibly and Shuler, 1973). FAST
provides a way to estimate the expected value and variance of the output variable and the
contribution of individual factors to this variance (Saltelli et al., 2000). FAST is a method based
on Fourier transformation of uncertain model parameters into a frequency domain. Through
Fourier transformation, a multi-dimensional model will be reduced into a single dimensional one.
16
Reliability Based Methods (FORM and SORM). First and second-order reliability
methods, also known as FORM and SORM, estimate the probability of an event under
consideration. These methods can provide the probability that an output exceeds a specific value,
also known as probability of failure (Karamchandani and Cornel, 1992; Lu et al., 1994; Hamed
et al., 1995). These methods are useful in uncertainty analysis of models that have a single
failure criterion.
3.2 Sensitivity Analysis
A series of seven sensitivity analysis methods that are of practical significance were
suggested in Volume 1 for further evaluation. These methods include Pearson and Spearman
correlation analyses, sample and rank regression analyses, analysis of variance (ANOVA),
Sobol’s method, and Fourier amplitude sensitivity test (FAST). These methods are briefly
explained in the following sections. Chapter 5 presents the results from evaluation and
comparison of these methods based on application to the simplified SHEDS-Pesticides model.
3.2.1 Correlation Analysis
Correlation coefficients, which are the special cases of regression-based analysis, are
typically useful methods for sensitivity analysis (Saltelli et al., 2000). The Pearson correlation
coefficient, for instance, can be used to characterize the degree of linear relationship between the
output values and sampled values of individual inputs. If the relationship between an input and
an output is nonlinear but monotonic Spearman correlation coefficients based upon rank
transformed values of an input and an output provides better performance compared to Pearson
correlation coefficients (Gibbons 1985, Siegel and Castellan 1988, and Kendall 1990). Neither
Pearson nor Spearman correlation coefficients can provide insight regarding possible interaction
17
effects between inputs. Resulting correlation coefficients can be used to rank the input values
based upon the magnitude of their influence on the output.
3.2.2 Linear Regression Analysis
Linear regression-based approaches for sensitivity analysis have the advantage that they
can evaluate the influence of various model inputs simultaneously compared to correlation-based
techniques (Saltelli et al., 2000). When each input in the regression model is standardized such
that the inputs are normally distributed about zero with a standard deviation of one, the
regression coefficients can provide insight into the relative significance of inputs (Devore and
Peck, 1996; Neter et al., 1996). Linear regression can also be performed using rank transformed
values of data instead of true values resulting in determination of rank regression coefficients. A
major limitation of regression-based approaches is their assumption of a known relationship
between model inputs and the output (e.g., linear, monotonically nonlinear, pre-specified
nonlinear terms, etc).
3.2.3 Analysis of Variance
If the relationships between the output and inputs cannot be precisely encoded into the
regression equations, there is a need for methods that do not assume any functional form for such
a relationship. One such approach is multi-factor ANOVA (Archer et al., 1997; Frey and Patil,
2002). ANOVA can quantify the relationship between the variation in one or more model inputs
and the changes in the mean of a model output (Neter et al., 1996). To perform ANOVA, the
model output values are binned as a function of their corresponding inputs. The influence of the
inputs can be evaluated by comparing the mean output value for each bin with that of the entire
dataset (Krishnaiah, 1981). ANOVA uses F tests to evaluate the statistical significance of input-
output relationship.
18
3.2.4 Sobol’s Method
Sobol’s method (Sobol, 1993; Saltelli et al., 2000) is a variance-based sensitivity analysis
technique based on Total Sensitivity Indices (TSI) that takes into account interaction effects
between inputs. The TSI of an input is defined as the sum of all of sensitivity indices involving
that input. The TSI involves first order effects as well as those of higher orders. Sobol’s method
is implemented by decomposing the input-output relationship into a summand of increasing
dimensionality. Solving the resulting equation to obtain sensitivity indices requires the use of
numerical integration techniques such as Monte Carlo, and is thus typically highly
computationally intensive. The total effects of inputs based on TSI do not provide a complete
characterization of the sensitivity. The interaction effect of an input (xi) with a specific input
(e.g., xi × xj) cannot be inferred from the TSI of the input (xi). However, the total effects are
much more reliable than the first-order effects in order to investigate the overall effect of each
single input on the output. Sobol’s method is model independent and can identify the
contribution of individual inputs to the output variance. However, Sobol’s method, in general, is
computationally intensive.
Sobol’s method is not readily available in commonly used statistical software packages.
Thus, an algorithm was prepared and coded in Matlab for application of Sobol’s method to the
case studies with the simplified SHEDS-Pesticides model. The algorithm is briefly explained in
Appendix B. This work represents the first time that Sobol’s method has been applied to a model
with multiple simulation time periods for multiple inputs. Thus, the basic algorithm prepared is
improved for application to the simplified SHEDS-Pesticides model with alternative sampling
strategies for inputs. The basic algorithm was verified based on results available from literature
(e.g., Saltelli 2002).
19
The test case used for verification of the Sobol’s algorithm was an analytic function
known as “Sobol’s g function”. This example was chosen because the cost of computation was
negligible and analytical answers were available for comparison with results based on Sobol’s
method.
The g function is a strongly non-monotonic, non-aditive function of k factors, xi, assumed
identically and uniformly distributed. The functional form of the “g function” can be given as:
∏= +
+−×=
k
i i
iik a
axxxxg
121 1
|24|),...,,( (3-1)
The importance of each input, xi, is driven by its associate coefficient ai. For ai = 0, the input is
important ( 2)(0 ≤≤ ii xg ). For, e.g., ai= 9 the input is non-important ( 1.1)(9.0 ≤≤ ii xg ), while
ai= 99 for the input can be considered as non-influent ( 01.1)(99.0 ≤≤ ii xg ). Further explanation
of the verification case study is given in Saltelli (2002).
The algorithm prepared for Sobol’s method was able to appropriately reproduce the
results given in Saltelli (2002). However, in some preliminary analyses with a simple linear
model that included the scale effect (i.e., inputs with substantially different mean values), the
algorithm showed slow convergence to the true sensitivity coefficients. However, when the
outputs from the algorithm were normalized, it was found that accurate results could be obtained
with a much smaller number of samples than implied by the raw results. The output values were
normalized using mean and standard deviations as:
y
yyyσ
)( −=′ (3-2)
where, y is the output value, y is the mean of the output values, and σy represents the standard
deviation of the output values.
20
In practice with our case studies with the simplified SHEDS-Pesticides model, the
algorithm developed for the Sobol’s method provided reasonable results in comparison to other
sensitivity analysis methods (See Chapter 5). Thus, a convergence problem is not suspected here.
3.2.5 Fourier Amplitude Sensitivity Test (FAST)
FAST (Cukier et al, 1973; Saltelli et al., 2000) is another variance-based global
sensitivity and uncertainty analysis procedure. FAST provides a way to estimate the expected
value and variance of the output variable and the contribution of individual inputs to this
variance. Similar to Sobol’s method, FAST provides first order indices representing the main
effects of inputs and TSI for model inputs. FAST is independent of the model structure. The
main difference between FAST and Sobol’s method is the approach by which the multi-
dimensional integrals are calculated. Whereas Sobol’s method uses a Monte Carlo integration
procedure, FAST uses a pattern search based on a sinusoidal function. The key advantage of
FAST is its capability in apportionment of the output variance to individual inputs in the model.
However, FAST is computationally complex for a model with a large number of inputs.
Similar to Sobol’s method, FAST is not readily available in commonly used statistical
software packages with exception of Simlab (Saltelli et al., 2004). This work represents the first
time that FAST has been applied to a model with multiple simulation time periods for multiple
inputs, and thus the case studies reported here is at the frontier of applicability of FAST.
Therefore, a modified algorithm for application of FAST was developed and coded in Matlab.
The most significant difference in this application of FAST versus those reported in the literature
is the need to deal with a much larger number of inputs.
Because the simplified SHEDS-Pesticides model has 30 time periods for each of the daily
inputs, and because the goal was to generate independent samples in each of the time periods,
21
there is a need for different frequencies for a given input variable for each time period (day) of
the simulation. Thus, although there are only 14 daily inputs, there is a need to generate 30 × 14
(i.e., 420) independent sets of samples. In addition, there are 11 monthly inputs. Thus, the total
number of independently sampled distributions is 431. Thus, there was a need to generate
additional frequencies beyond the set of 50 incommensurate frequencies available in literature.
This algorithm is briefly discussed in Appendix B. Basic algorithm of FAST was verified with
the same set of test cases used for verification of Sobol’s method. The algorithm prepared for
FAST was able to appropriately reproduce the results given in Saltelli (2002).
This chapter has briefly summarized methods that are discussed in more detail in Volume
1. Because the focus of this work is on evaluation of methods for sensitivity analysis, and
because there are not readily available software packages for Sobols’ and FAST methods,
algorithms were developed and coded for these two methods based upon information in the
literature. In the case of FAST, the algorithm had to be modified in a manner that allowed for
more inputs than the limit of 50 that has been used for years, but possibly could introduce a
dependency structure in the simulation that might compromise the case study results. However,
in order to make a preliminary assessment of whether FAST could be appropriate in the future,
the current version of the modified algorithm was applied to the simplified SHEDS model test
bed, along with the other methods.
23
4. DEFINING CASE SCENARIOS FOR SENSITIVITY ANALYSIS
The objective of this chapter is to define a case study scenario as the basis for performing
sensitivity analysis on the simplified SHEDS-Pesticides model. A scenario is a set of
assumptions about the nature of the problem to be analyzed (Cullen and Frey, 1999). These
assumptions may be based upon recommendations from field experts, stakeholders, risk
managers, or combinations of all three.
The definition of the most relevant or important scenarios is especially crucial in
situations for which there are limitations of time and other resources with respect to performing
sensitivity analysis. Thus, it is important to identify scenarios that are the highest priority for
evaluation. A well defined case scenario will help concentrate the sensitivity analysis on the
areas that are of more interest to risk managers and decision makers.
Figure 4-1 shows a schematic diagram of the case study scenario components for
sensitivity analysis of the simplified SHEDS-Pesticides model. This figure shows that issues
such as susceptible subpopulations, pathways of exposure, and time scale of the model
simulation should be clearly specified in the case scenario. Furthermore, the probabilistic
dimensions that are intended to be the focus of sensitivity analysis should be selected in a case
scenario. The methodology for sensitivity analysis is not dependent on these factors. Thus, for
purpose of demonstrating methods, it is not necessary to consider all possible scenarios.
Furthermore, in order to have meaningful outcomes from the sensitivity analysis that can be used
by risk managers, it is useful to define specific case scenarios relevant to the model scope that
are of policy interest.
24
Figure 4-1. Components of a Scenario for Sensitivity Analysis of the Simplified SHEDS-
Pesticides Model.
In the following sections, the major components of the scenario defined for sensitivity
analysis are explained. These components are classified into four categories including: (1)
probabilistic dimensions; (2) susceptible subpopulation; (3) identification of pathways of
interest; and (4) time scales of the model simulation. Sections 4.1 through 4.4 provide
discussions for these topics, respectively. For the defined scenario, three ways of summarizing
the results as a function of time are considered. Thus, there are multiple scenarios with respect to
interpretation of the temporal aspects. However, these scenarios are based upon one set of
assumptions regarding the other aspects.
4.1 Probabilistic Dimensions
The probabilistic dimension of a scenario indicates whether the assessment will explicitly
incorporate variability, uncertainty, or both. A scenario will typically include variability in
exposures among different members of a population, unless the scenario is for a single
individual. Regardless of whether the scenario is for a population or for a single individual, there
will typically be uncertainty in the inputs to a model. Therefore, as a part of defining the
scenario, it is important to define the probabilistic dimension of the scenario. Furthermore, there
Scenario
Probabilistic Dimension
Susceptible Subpopulation
Time Scale
Exposure Pathway
25
are alternative methods for dealing with variability and uncertainty. The choice of an appropriate
method should be made taking into account the assessment objectives, the data quality
objectives, the availability of data, and the importance of the assessment. Because the simplified
SHEDS-Pesticides model has a one-dimensional probabilistic framework considering variability
in inputs, variability-only analysis is selected for the probabilistic dimension of sensitivity
analysis.
The purpose of an analysis of variability-only is to quantify inter-individual variability in
exposure and risk. Such an analysis is typically predicated on the assumption that the range of
variability is much larger than the range of uncertainty; therefore, a judgment is made that
uncertainty can be neglected. The appropriateness of this assumption will depend upon the
specific problem and the objectives of the analysis. Variability can include controllable or
explainable sources of variation (e.g., differences in time spent in pesticide contaminated area
among members of a population) or stochastic sources of variability (e.g., differences in fraction
of chemical available for transfer on a daily basis).
4.2 Susceptible Subpopulation
The scope of the risk assessment is typically on specific population groups in which the
risk of adverse effect due to exposure to the hazard is expected to be significant. For a particular
pollutant, there may be multiple susceptible subpopulations because different groups of people
may be exposed to that particular type of pollutant. Susceptible subpopulations can be
incorporated in the model using alternative dose-response relationships. Furthermore, exposed
subpopulations could be differentiated by age classes with respect to activity patterns, weight,
and inhalation rate.
26
For the purpose of evaluation of selected sensitivity analysis method, children between 5
and 10 are considered as the population of interest in the simplified SHEDS-Pesticides model.
Typically, children between 5 and 10 spend considerable amount of time outdoor in play
grounds, and hence, have high potentials for exposure to pesticides. The information regarding
body height and weight for each member of population is provided with consideration of the
selected age range. As a caveat, because sensitivity analysis is therefore focused on a subset of
the general population, insights regarding key sources of variability may not be representative of
other groups.
4.3 Identification of Pathway of Interest
In order to adequately characterize total exposures, a scenario may need to consider
multiple pathways of exposure. The simplified SHEDS-Pesticides model considers three
different exposure pathways including inhalation, dermal (via body and hand), and ingestion.
Total exposure represents the aggregation of exposure from all three pathways. Figure 4-2 shows
probability distribution of average exposure from each of the three pathways and for the average
total exposure in 30 days of model simulation. The results of the different exposure pathways in
this figure are based upon the simplified SHEDS-Pesticide model and input assumptions
explained in Chapter 2 and Appendix A. The ranking of three exposure pathways with respect to
the magnitude of the average monthly exposure is: (1) dermal; (2) ingestion; and (3) inhalation
pathway. For the purpose of sensitivity analysis and evaluation of selected methods, total
exposure is considered as the output of interest.
4.4 Time Scales of the Model Simulation
A risk assessment scenario may include a temporal dimension. The temporal
considerations for a scenario typically include: (1) the time for each major step in the exposure
27
0
0.2
0.4
0.6
0.8
1
0.001 0.1 10 1000
Mean Exposure in 30 Days (µg/kg)
Cum
ulat
ive
Pro
babi
lityInhalation ExposureIngestion ExposureDermal ExposureTotal Exposure
Figure 4-2. Probability Distribution of Different Average Exposure Pathways in 30 Days of Model Simulation.
episode, in order to estimate the exposure at each step; (2) the activity patterns of individuals
with regard to frequency of exposure to particular types of pollutant; (3) “temporal dynamics”
effects, whether at a short time scale (e.g., daily, weekly) or a longer scale (e.g., monthly,
seasonal, annual); and (4) the time period associated with occurrence of illness as a result of one
or more exposures (Frey et al., 2004). In the simplified SHEDS-Pesticides model, inputs have
different sampling frequencies with respect to time. Some inputs such as body weight or amount
of applied pesticide are only sampled once for each individual during the entire simulation time
of the model (e.g., 30 days). In contrast, some other inputs such as inhalation time and body
washing removal efficiency are sampled in shorter time steps for each individual (i.e., daily).
Three analysis scenarios are defined in order to evaluate the implications of different time scales,
including: (1) Scenario I - the daily total exposures; (2) Scenario II - incremental change in daily
exposure representing the rate of change of exposures from one day to the next; and (3) and
Scenario III - longer-term cumulative exposures.
28
Figure 4-3 shows variation of total exposure for selected random individuals in 30 days
of model simulation. The results are based upon the simplified SHEDS-Pesticide model and
input assumptions explained in Chapter 2 and Appendix A. Daily exposure for each individual is
the direct output from the model at each time step of the simulation, )(ty , where y is the total
exposure from the model and t represents the time step of the model simulation. Incremental
change in daily exposure represents the difference between the exposure at the current time step
and the previous time step (i.e., )1()( −− tt yy ). Cumulative exposure is sum of exposures from day
1 to the current time step of the model simulation (i.e.,∑=
t
iiy
1)( ).
Table 4-1 summarizes an example of the output of the model for the three time scales of
the scenario. Results are given for the individual ‘a’ shown in Figure 4-3. Incremental change in
daily total exposure was either negative or positive at different time steps of the model
simulation, while daily and cumulative exposures were always positive. For example, because
daily total exposures in the first and second days were 48.9 and 25.1 µg/m3, respectively, the
incremental change in daily total exposure in the second day had a negative value of -23.8
µg/m3. In contrast, in the third day, the incremental change in daily total exposure had a positive
value of 13.8 µg/m3 due to increase in the daily exposure from 25.1 to 38.9 µg/m3 between the
second and third days, respectively.
29
(a) Scenario I: Daily Exposure
0
20
40
60
80
100
1 6 11 16 21 26 31
Time (day)
Tota
l Exp
osur
e ( µ
g/kg
)Individual a Individual b Individual c Individual d
(b) Scenario II: Incremental Change in Daily Exposure
-60
-40
-20
0
20
40
60
1 6 11 16 21 26 31
Time (day)
Tota
l Exp
osur
e ( µ
g/kg
)
(c) Scenario III: Cumulative Exposure
0
100
200
300
400
500
600
700
800
1 6 11 16 21 26 31
Time (day)
Exp
osur
e ( µ
g/kg
)
Figure 4-3. Example of Probabilistic Results for Variation in the Total Exposure for Selected Random Individuals: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in
Daily Exposure; and (c) Scenario III-Cumulative Exposure.
30
Table 4-1. Example of Probabilistic Results for Daily, Incremental Change in Daily, and Cumulative Total Exposure for the Random Individual ‘a’ shown in Figure 4-3
Time [day]
Scenario I: Daily Total Exposure
[µg/m3]
Scenario II: Incremental Change in Daily
Total Exposure [µg/m3]
Scenario III: Cumulative Total
Exposure [µg/m3]
1 48.9 48.9 (a) 48.9 2 25.1 -23.8 74.0 3 38.9 13.8 112.9 4 40.9 2.0 153.8 5 77.2 36.3 231.1 6 61.9 -15.3 293.0 7 32.8 -29.1 325.8 8 48.5 15.7 374.3 9 35.1 -13.4 409.4 10 14.2 -20.9 423.6 11 23.4 9.2 447 12 20.4 -3.0 467.4 13 28.0 7.6 495.4 14 27.0 -1.0 522.4 15 20.5 -6.5 542.9 16 23.8 3.3 566.7 17 29.0 5.2 595.7 18 21.6 -7.4 617.3 19 21.5 -0.1 638.8 20 14.8 -6.7 653.6 21 10.0 -4.8 663.6 22 11.1 1.1 674.7 23 8.6 -2.5 683.3 24 6.8 -1.8 690.1 25 3.0 -3.8 693.1 26 3.3 0.3 696.4 27 4.8 1.5 701.2 28 6.4 1.6 707.6 29 3.0 -3.4 710.6 30 2.0 -1.0 712.6
a The incremental change in daily total exposure at Day 1 is assumed to be equal to the total daily exposure in that day
31
5. RESULTS OF SENSITIVITY ANALYSIS FOR CASE STUDIES
The objective of this chapter is to present the results from applying selected sensitivity
analysis methods to the case scenario defined in Chapter 4. Section 5.1 provides a brief
discussion regarding the model application, including general algorithm used for application of
the selected sensitivity analysis methods, three different temporal scenarios considered in the
analysis, and the sample size for methods based on Monte Carlo simulation. The specific
methods evaluated here include Pearson and Spearman correlation analyses (Section 5.2), linear
sample-based and rank-based regression (Section 5.3), ANOVA (Section 5.4), Sobol’s method
(Section 5.5), and FAST (Section 5.6). These methods are briefly discussed in Chapter 3 and
extensively explained in Volume 1. Section 5.7 presents a comparison of results based on
different sensitivity analysis methods.
5.1 Model Application
The simplified SHEDS model documented in Chapter 3 was used as a testbed for
development of quantitative sensitivity analysis results for each of the seven selected sensitivity
analysis methods. As described in Chapter 4, the simplified model is applied to an exposure
scenario involving residential exposure to pesticides. The pesticide is applied at the beginning of
a month and exposures are estimated on both a daily and cumulative basis for all exposure
pathways.
The application of the seven selected sensitivity analysis methods involves sensitivity
analysis for each time step (day) of the model simulation. For the six sampling-based methods,
which include all of the selected methods except FAST, probability distributions representing
inter-individual variability in the inputs were propagated through the simplified SHEDS-
Pesticide model using Monte Carlo simulations with a sample size of 10,000. For inputs that had
32
a monthly sampling strategy, and hence, did not vary on a day-to-day basis, one value per
individual per month was sampled, for a total of 10,000 samples. These values were used for all
time steps of the model simulation. In contrast, for inputs that had a daily sampling strategy, and
hence, varied on a day-today basis, 30 values per individual were sampled with a different value
in each time step of the model simulation. Thus, there are a total of 300,000 samples for those
inputs that vary daily and also among individuals.
The sample size for FAST was assigned based on the number of inputs and the maximum
frequency associated with inputs. Further explanations for selecting the sample size for FAST are
given in Volume 1.
Figure 5-1 shows a schematic diagram of the general algorithm for application of the
selected sensitivity analysis methods to the simplified SHEDS-Pesticide model. Each dataset in
this figure includes randomly sampled values from variability distributions of inputs at a given
day for sampling-based methods or values generated from domains of inputs using the
transformation functions in FAST and the corresponding model output values. Output values
were based on three temporal scenarios introduced in Chapter 4, which are: (1) Scenario I - daily
total exposure; (2) Scenario II - incremental change in daily total exposure; and (3) Scenario III -
cumulative total exposure. Sensitivity analysis was applied separately to each dataset. At each
time step, the key sources of variability were identified. This process was repeated n times to
arrive at different sensitivity rankings for inputs in which n refers to the total number of time
steps considered in the model simulation (i.e., 30 days). A ranking represents the comparative
order of importance of each input on a given day when the inputs are sorted according to their
sensitivity indices. A rank of 1 refers to the most important input. As the numerical value of the
33
Figure 5-1. Schematic Algorithm for Application of Selected Sensitivity Analysis Methods to the Simplified SHEDS-Pesticides Model.
rank increases, the importance decreases, the rank is said to be worse. The variation of rankings
when comparing different time steps were used to assess whether the sensitivity of inputs
changed at different times based on different types of model responses to each input and on
possible differences in the importance of inputs with sampling strategies of monthly versus daily
at different times of the month.
5.2 Correlation Analysis
This section presents the results based upon the use of Pearson and Spearman correlation
coefficients to characterize the sensitivity of the selected model outputs for temporal Scenarios I,
II, and III to each of the inputs of the model. For discussion purposes, this section focuses on
providing specific details for selected inputs, followed by an overall summary of results for all of
the model inputs. The results presented in this section include: Pearson correlation coefficients
(PCC) and Spearman correlation coefficients (SCC) for selected inputs; changes in rankings of
Dataset 1
Dataset 2
Dataset n
Sensitivity Analysis
Sensitivity Analysis
Sensitivity Analysis
Ranking 1
Ranking 2
Ranking n
Summary of Sensitivity
Analysis Results
n : Number of time steps for model simulation (e.g., n=30)
34
selected inputs with respect to time; and mean ranks and ranges of ranks during the 30 days of
model simulation for all inputs. The inputs selected for detailed discussion include residue decay
rate (DR), probability of washing body (PWB), and fraction of chemicals available for transfer
(FTR). These inputs are further explained in Section 2.3. These inputs were selected because they
illustrate the following key issues: (1) they have different sampling strategies; (2) they have
different patterns with respect to sensitivity indices and rankings with respect to time; and (3)
typically, they were among the most sensitive inputs.
5.2.1 Correlation Coefficients Results
Regardless of the frequency with which inputs were sampled, correlation coefficients
were estimated at each day for all inputs for each of the three temporal scenarios, and hence, 30
sets of Pearson and Spearman correlation coefficients were estimated. This is because there
could be changes over time in sensitivity of inputs sampled on a monthly basis depending on the
sensitivity of phenomena that occurred on a daily basis. Typically, both Pearson and Spearman
correlation coefficients for each input varied in time in the three temporal scenarios. For inputs
with a monthly sampling strategy, two distinct patterns with respect to the change in the absolute
value of the correlation coefficients over time were identified: (1) monotonic decrease; and (2)
monotonic increase. However, for inputs with a daily sampling strategy, correlation coefficients
typically declined in time. Examples are provided to illustrate and explain these patterns.
Figure 5-2 shows variation in absolute values of PCC and SCC and corresponding 95%
confidence intervals for the residue decay rate (DR) with respect to time. The coefficients are
shown for the three temporal scenarios. DR has a monthly sampling strategy and provides an
example in which the input tends to have a higher correlation coefficient later in the month.
35
(b) Scenario II: Incremental Change in Daily Exposure
0
0.25
0.5
0.75
1
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
(c) Scenario III: Cumulative Exposure
0
0.25
0.5
0.75
1
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
Figure 5-2. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and Spearman Correlation Coefficients for Re sidue Decay Rate as an Input with Monthly Sampling
Frequency for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
(a) Scenario I: Daily Exposure
0
0.25
0.5
0.75
1
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
95% Confidence Interval of the Coefficient
36
Figure 5-2 (a) shows the results for Scenario I. DR has a substantially higher correlation with
daily total exposure later in the month. For example, SCC and PCC had low values of 0.08 and
0.05, respectively, in the first day. However, these coefficients increased to 0.87 and 0.37,
respectively, by the end of the month. Increase in the magnitude of correlation coefficients, and
hence, the sensitivity of DR in later time steps of the model simulation can be attributed to the
functional form of the simplified SHEDS-Pesticides model. DR is used in the model when
chemical residues at target and non-target surfaces are estimated on a daily basis (See Equations
13-a and 13-b in Appendix A). However, because the effect of DR is raised to the power i, where
i represents the day for which the effect of residue decay rate is estimated, DR will have higher
contribution to variation of the model output later in the month and with the increase in the value
of i. Thus, although daily exposures were higher within the first few days of the month (See
Figure 4-3), DR was more correlated to the model output later in the month. However, because
the daily exposures typically decrease as more time elapses from the time of pesticide
application, the higher daily correlation late in the month for DR may not be of as much
significance when compared with other inputs that have high correlations earlier in the month,
when total daily exposures are higher.
Figure 5-2 (b) show the results for Scenario II for the change in PCC and SCC of DR with
respect to the time. Typically, the change in magnitude of correlation coefficients for this
temporal scenario follows a similar pattern compared to Scenario I. However, DR typically had
lower correlation with the incremental change in the daily exposures than with the total daily
exposures. The likely reason is that although the residue decay rate implies that there is a
continuous change over time in the amount of pesticide residue, the numerical value of DR itself
does not change from one day to the next. Thus, any value of DR other than zero will imply
37
some incremental change from one day to the next, but other inputs (and particularly those that
vary on a daily basis) are responsible for larger incremental changes. As noted in Chapter 4, and
in the example of Table 4-1, incremental changes can be positive or negative, and thus are
inherently “noisier” than the results from the other two temporal scenarios. For example, the
cumulative exposure in Scenario III must always increase monotonically, and typically the
marginal increase becomes smaller with time, and thus Scenario III would typically be expected
to produce the most stable results for sensitivity indices when comparing one day to the next,
whereas Scenario II would be expected to produce the least stable results.
Figure 5-2 (c) shows the results for Scenario III. Absolute values of PCC and SCC for DR
along with their 95% confidence intervals are shown in 30 days of model simulation. Similar to
the previous temporal scenarios, the magnitude of correlation coefficients increased in later time
steps of the model simulations. However, the magnitude of coefficients implies that DR had
lower correlation with the cumulative exposure at each day compared to the daily exposure in the
same day. The cumulative exposure for each individual at a selected day not only depends on the
value of DR in the same day, but also it relies on the values that DR held on previous days. As
shown in Figure 5-2(a), the daily exposures have a small correlation with DR early in the month.
However, especially in the first several days, the magnitude of the daily exposures is typically
the largest. By mid month, most of the cumulative exposure has occurred, and the marginal
increase in cumulative exposure becomes smaller, on average, on a daily basis. Thus, although
DR is highly correlated with the decreasing values of daily exposure in the later days of the
month, these daily exposures contribute relatively little to the cumulative total. For example, by
the 20th day, there is little perceptible increase in the magnitude of the correlation coefficients
38
shown in Figure 5-2(c), which is indicative of little or no marginal impact of DR late in the
month.
Comparing Pearson and Spearman coefficients for the three temporal scenarios indicates
that rank transforming the data typically increased the magnitude of the correlation coefficients
for DR. The highest increase in the magnitude of the coefficients was for Scenario I. DR has a
nonlinear relationship with daily exposure values. Because of the nonlinearity, the PCC is not a
good measure of sensitivity. The relatively low values of 0.30 for the PCCs in the last half of the
month do not appropriately quantify the strength of the relationship between daily exposure and
the selected input. However, although nonlinear, the relationship between daily exposure and DR
is monotonic. Therefore, the SCC provides a much better measure of the sensitivity of the output
to DR. as reflected in the larger magnitude of the correlation that is achieved based upon rank
transformation of the data.
For Scenario II, rank transforming the data did not improve the correlation between the
output values and DR. When the correlation is small for PCC, it is also small for SCC. The
differences between the two only become apparent when the correlations become larger.
Because Scenario II generally has low correlations, it matters less as to what type of correlation
coefficient is used to quantify the magnitude of the correlation. In other words, both methods
seem equally capable of indicating situations in which the correlations are very low.
For Scenario III, rank transforming data had a minimal effect on improving the
magnitude of correlation coefficients. The cumulative exposure is based upon the summation of
exposures over a given number days, and thus has a strong linear (additive) aspect. Although the
daily estimate of exposure has a nonlinear relationship with DR, the cumulative exposure is an
additive combination of multiple days. Furthermore, the cumulative exposure is most sensitive
39
to those days in which the highest magnitude of daily exposures occurred. The highest daily
exposures were early in the month. Early in the month, the residue decay rate was typically not
high enough to cause a large change in the amount of pesticide residue, and therefore the
behavior of the model of daily exposures to DR is approximately linear, especially in the first five
days. This is corroborated by the similarity of the PCC and SCC values for daily exposures in
the first five days. By the 10th to 15th day, the cumulative exposure is quite large compared to
that on the 30th day, and thus is most highly influenced by the effect of residue decay rate in the
first half of the month. Between the 5th and 15th day, the daily exposure response to DR becomes
increasingly nonlinear, which causes some nonlinearity in the response of the cumulative
exposure as well. However, the effect of the nonlinearity in cumulative exposure in the middle
part of the month is somewhat attenuated because of the stronger influence of the more linear
effect in the first five days.
Confidence intervals given for the correlation coefficients provide insight with respect to
statistically significant differences between coefficients. Typically, correlation coefficients based
on Pearson and Spearman methods did not statistically differ in the first few days of the model
simulation for the three temporal scenarios. However, typically later in the month, there were
statistically significant differences between PCC and SCC, typically because of nonlinearities in
the model as previously discussed. Furthermore, the confidence intervals can be used to assess
whether changes from one time period to another are significant. For example, for Scenario I,
there is a significance difference in sensitivity on the 30th day versus the 15th day based on SCCs,
but the SCCs during the last five days are approximately the same. Likewise, the SCCs are
approximately the same for the last 15 days for Scenario III.
40
Figure 5-3 shows variation in absolute values of PCC and SCC and corresponding 95%
confidence intervals for probability of washing body (PWB) with respect to time. PWB has a
monthly sampling strategy and provides an example in which the highest correlations for
Scenario I occurs early in the month. The magnitude of the correlation coefficients is relatively
modest, ranging from approximately 0.1 to 0.3. However, the association is significantly
stronger at the 5th day when compared to later in the month, such as after the 15th day. The
reason for a peak in sensitivity is that as time elapses, there is a change in the order of
importance among several inputs. For example, in the first few days of the month, the residue
decay rate is not one of the most important inputs. However, as time elapses, the variation in the
amount of pesticide residue remaining has a stronger relationship to residue decay rate. Thus, as
one input increases in importance, and therefore changes in rank toward ranks closer to one,
other inputs must simultaneously be displaced in terms of their importance. Thus, there is a
temporal dynamic in sensitivity.
The larger magnitude of the SCC compared to the PCC suggests that there is a moderate
nonlinear but monotonic association with this input. Furthermore, when compared to DR, PWB
has a higher magnitude of the SCC in the first several days, and thus has a more sensitive
ranking early in the month, when daily exposures tend to be the highest. For example, PWB has a
rank of one, whereas DR has a rank of six on Day 5. Over time, these rankings change, with DR
having an improved rank (closer to 1) later in the month, and PWB simultaneously having a
worsening rank. Of course, as noted earlier, the magnitude of daily exposures is highest early in
the month, and thus it is more significant that PWB is highly ranked (closest to 1) early in the
month.
41
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
Figure 5-3. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and
Spearman Correlation Coefficients for Probability of Washing Body as an Input with Monthly Sampling Frequency for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in
Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
95% Confidence Interval of the Coefficient
42
Early in the month, when PWB is one of the most important inputs with respect to daily
exposure, it tends to have a higher sensitivity with respect to the incremental change in daily
exposures, as shown in Figure 5-3(b). However, within the first few days, the magnitude of both
the SCC and PCC decreases significantly, and toward the end of the month these coefficients are
either not statistically significant or they are not of any practical significance. In contrast, as will
be shown in the next example, inputs that vary on a daily basis tend to have more influence on
the incremental amount of change from one day to the next late in the month, but not necessarily
on the late-in-the-month magnitude of the daily exposures or especially on the cumulative
exposures. Thus, the results for inputs that have a monthly sampling frequency with respect to
Scenario II are based in part on a competition with those inputs that vary on a daily frequency.
PWB increases in importance with respect to cumulative exposure in Scenario III, as
shown in Figure 5-3(c), during the first five days of the month. However, from Day 5 to Day 30,
there is no significant change in either the PCC or the SCC. This result is attributable to two
concurrent causes: (1) this input becomes less important on a daily basis later in the month; and
(2) the daily exposures become much smaller later in the month. Compared to DR, the main
difference is that PWB decreases in importance on a daily basis, whereas DR increased in
importance on a daily basis, most notably in days 5 to 10. Therefore, DR had some impact on an
increased cumulative sensitivity during that time. In contrast, PWB simultaneously declined in
importance as the magnitude of daily exposures decreased, and its impact on cumulative
exposure essentially remains “frozen” after the fifth day.
Typically, correlation coefficients based on the ranked transformed data were larger than
those based on the sampled data for Scenarios I and III, and there were small but statistically
significant differences especially for Scenario III. The similarity in the two types of correlation
43
coefficients may imply that if there is a nonlinearity effect, or some other type of interaction
effect, it is relatively small. For Scenario II, there was typically no significant difference
between the two types of correlation coefficients, but the magnitudes were typically small for
both.
Figure 5-4 shows an example of change in correlation coefficients for the fraction of
chemicals available for transfer (FTR). This input has a daily sampling frequency. For Scenario
I, the correlations were significantly higher on the first day than on the second day, and the
magnitudes of both the SCC and PCC decreased approximately monotically with time. The
reason for the declining importance of this input, and of other daily inputs not shown here, is
further described later in this section.
Although FTR rapidly declined as an influence on the magnitude of the daily exposure as
time elapsed, it remains one of the more important reasons for incremental changes in exposure
from one day to the next. Unlike the inputs that are sampled only once per month, the value of
FTR changes each day, which in turn can cause incremental changes in the daily exposures. As
illustrated in Table 4-1, the incremental changes in daily exposures tend to decrease as time
elapses. Thus, it is possible that an input could be highly important with respect to incremental
changes, but be of less importance to inter-individual variation in the magnitude of the daily
exposure. This is because some of the variation in the daily exposure magnitude on a given day
may be approximately unchanged from one day to the next, such as because of the influence of
inputs with a monthly sampling frequency. Typically, it is expected that inputs that vary with a
high frequency would be more important with respect to short-term incremental changes but
would not necessarily have a strong impact on the underlying causes of variability from one day
to the next.
44
(b) Scenario II: Incremental Change In Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
Figure 5-4. Absolute Values and Corresponding 95% Confidence Intervals of Pearson and
Spearman Correlation Coefficients for Fraction of Chemicals Available for Transfer as an Input with a Daily Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental
Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30
Time (day)
|Cor
rela
tion
Coe
ffici
ent|
PearsonSpearman
95% Confidence Interval of the Coefficient
45
Figure 5-4(c) illustrates another typical pattern for inputs that vary on a daily basis. For
an individual day, such inputs can be quite important, as indicated in the result for the first day.
The correlations here are approximately 0.4 for PCC and 0.5 for SCC, which are relatively high.
By the second day, the correlation is only approximately 0.2 for either type of correlation, and by
the 6th day the correlations are statistically insignificant in both cases. A key assumption in the
simulation is that this input has independent values from one day to the next. Thus, it is possible
that the input could have a high value on one day, and a low value on the next day, or vice versa.
When comparing Scenario I and III, it is apparent that this input is important on the first day, and
declines significantly in importance with respect to daily exposures. Simultaneously, the
magnitude of daily exposures typically decreases with time. Thus, the cumulative effect tends to
decrease. However, unlike the situation for inputs that remain constant during the month, an
additional factor is likely to be important.
The additional factor is the effect of averaging over the course of many days for those
inputs that vary on a daily basis. This effect is easiest to identify in a purely linear model. As an
input is sampled independently from one day to the next, its values will fluctuate randomly
among high and low values according to the frequency distribution from which daily values are
generated. However, by the end of the month, the overall effect of such an input will be based
upon the sampling distribution of the mean. The sampling distribution of the mean has a much
smaller variance than does the frequency distribution of the daily variability. Thus, by the end of
the month, the effective range of variation for a given individual tends to narrow and converge
with respect to the mean. Furthermore, if all individuals are assumed to have independent
random samples from the same frequency distribution, as is the case here, then the overall effect
is to arrive at a sampling distribution of the mean based upon 300,000 samples. The standard
46
error of a sampling distribution with such a large number of samples is quite small – in this case,
less than approximately 0.2 percent of the population standard deviation. Thus, daily inputs
sampled in this way are destined to become unimportant with respect to cumulative exposure
unless the model is highly nonlinear or non-independent sampling schemes are used.
For Scenarios I and III, the SCC and PCC correlation coefficients had similar
magnitudes, and their confidence intervals typically overlapped, except on the first day. This is
consistent with observations for the previously discussed inputs that these two types of
correlations produce similar numerically values when correlations are small, but they tend to
differ when correlations are large and presumably when there is a nonlinearity characteristic to
the model. For Scenario II, for which the SCCs tend to be consistently at a value of
approximately 0.5, there is a more pronounced difference compared to the PCCs. In fact, not
only the numerical values but the trends appear to be different. For example, while the SCCs are
approximately constant over all days of the month, the PCCs appear to decrease monotonically
toward values that, while statistically significant, imply a weak correlation. A possible reason
for this difference is an overall change in the behavior of the model later in the month. As noted
earlier in the discussion of residue decay rate, the model becomes more sensitive to the residue
decay rate on a daily basis. The model has a nonlinear response to residue decay rate that
becomes more pronounced later in the month. Thus, there may be an interaction effect
associated with this source of nonlinearity, which could explain why the SCCs and PCCs for
other inputs, such as FTR, diverge more sharply later in the month.
A detailed examination of the temporal profile of sensitivity results for selected inputs
illustrates the complexity of the temporal dynamic of sensitivity, the competition among inputs,
47
and the changing response of the model as different components or aspects (e.g., nonlinearity) of
the model become more dominant.
5.2.2 Rankings for Selected Inputs
At each time step, inputs were ranked ordered based on the relative magnitude of
correlation coefficients. A rank of one is assigned to an input with highest correlation coefficient,
and the largest numerical value of rank was assigned to the input with the least importance.
However, inputs that have ranks closest to one (that are highly important) are typically described
as inputs that are “highly ranked” or that have “higher” or “high” ranks, whereas inputs with
large numerical values of rank (implying lack of importance) are typically described as having
“low” rank. The rank for each input typically changed with respect to time. Some inputs had
higher ranks in earlier time steps of the model simulation, while their sensitivity declined later in
time. In contrast, some other inputs were typically unimportant in earlier time steps, while their
sensitivity improved later in time. An example for each case is provided. Examples provided in
the following are based on those shown in Figures 5-2 and 5-4, for residue decay rate (DR) and
fraction of chemicals available for transfer (FTR).
Figure 5-5 shows the variation in rank of DR with respect to time for three temporal
scenarios. The ranks are based on PCC and SCC in Figures 5-5 (a) and 5-5 (b), respectively. DR
is an example of an input that had less sensitivity compared to other inputs earlier in the month,
while its sensitivity increased later in the month. The most significant findings are that DR
initially was of only moderate importance to both daily and cumulative exposures early in the
month, but became more important later in the month for reasons previously discussed. Late in
the month, DR was typically the top ranked input. Of course, the magnitude of exposure
decreases later in the month, and thus the daily exposures become less of a factor in terms of
48
(a) Pearson Correlation
0
510
1520
25
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Spearman Correlation
0
5
101520
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-5. Variation in Rank of Residue Decay Rate as an Input with Monthly Sampling Strategy with Respect to Time Based on:
(a) Pearson; and (b) Spearman Correlation Coefficients.
Lower Rank Implies Greater Sensitivity
49
influence on the cumulative exposure. Nonetheless, in terms of overall impact on cumulative
exposure, DR is the most important input by Day 6 and at the end of the month. The rankings for
cumulative exposure at the end of the month are likely to be the most indicative of the overall
importance of various inputs with respect to cumulative exposure.
The rank of DR with respect to incremental changes in exposure (Scenario II) has an
interesting pattern. Initially, DR is of moderate or minor importance, with a rank of
approximately 10. By the seventh day, the rank worsens, and DR becomes one of the least
important inputs. By the end of the month, however, DR is one of the most important inputs.
However, the distinction between a rank in the top five versus a rank of approximately 10 is
more pronounced than that between a rank of 10 and one of the lowest ranks. In other words,
there is more difference in sensitivity between a top rank and moderate rank than between a
moderate rank and a low rank. Thus, in approximate terms, DR is not a particularly important
input with respect to incremental change until late in the month. As noted in previous text, the
model becomes more strongly sensitive to DR later in the month, and the model response to DR is
nonlinear. Thus, it appears to be the case that incremental changes in exposure late in the month
are highly influenced by the incremental change in pesticide residue remaining, which in turn is
highly influenced by the residue decay rate.
Figure 5-6 shows the variation in rank of fraction of chemicals available for transfer (FTR)
with respect to time for the three temporal scenarios based on PCC and SCC. FTR is an example
of an input that had relatively higher sensitivity earlier in the month, while its sensitivity declined
later in the month for Scenarios I and III. Furthermore, unlike DR, FTR has a daily sampling
frequency. On the first day, FTR is the most important input for all three temporal scenarios. By
day 2 or 3, however, FTR begins to decline in its rank order, especially with respect to the
50
(a) Pearson Correlation
0
510152025
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Spearman Correlation
0
5
101520
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-6. Variation in Rank of Fraction of Chemical Available for Transfer as an Input with Daily Sampling Strategy with Respect
to Time Based on: (a) Pearson; and (b) Spearman Correlation Coefficients.
Lower Rank Implies Greater Sensitivity
51
cumulative exposure. By approximately the 5th day, FTR reaches a consistent ranking in Scenario
I of approximately the fifth most important input. However, by the 10th day, FTR becomes one of
the least important inputs with respect to cumulative exposure. Although the ranking in Scenario
III appears to be “noisy,” with what appear to be large changes from day-to-day, rankings that
are higher in numerical value than approximately 15 are associated with inputs that have either
minor or insignificant sensitivity. Thus, large changes in rank order among insignificant inputs
does not imply any significant change in importance. In contrast, small changes in rank order
among important inputs can imply a significant change in importance. Furthermore, the fact that
FTR is unimportant on the 30th day for the cumulative exposure suggests that this input has little
overall effect on cumulative exposure. If one is concerned with very short-term acute exposures,
the fact that this input is the most important on the first day, when exposures tend to be high,
would be of significance. However, if one is concerned with average or cumulative exposures
over many days, then this input is not likely to be of much importance.
As noted earlier in the discussion of the magnitude of the correlation coefficients, FTR is
consistently the most important input with respect to incremental change in daily exposures
(Scenario II). This may be useful information for understanding what factors govern the rate at
which daily exposures are changing, and thus understanding the behavior of the model.
However, the incremental changes from one day to the next may not be the dominant cause for
variation in exposures among individuals in a given day nor on the cumulative exposure. For
example, DR has a persistent effect on inter-individual variability in daily and cumulative
exposures even though from one day to the next it does not cause a large change in exposure for
any given individual. Thus, DR would likely be considered to be a more significant or important
input than FTR.
52
5.2.3 Comparison of Mean Ranks
This section presents the mean rank associated with each input within the time period of
the model simulation. The mean rank represents the arithmetic average of ranks for each input.
The range of ranks also is given for each input. The range of ranks represents minimum and
maximum ranks for the input within 30 days of model simulation. However, a rank of 1 on day
one might be much more significant than a rank of 1 on day 30, because the daily exposure on
day one is much higher than on day 30, and the effect on the cumulative exposure is more
pronounced early in the month than later in the month. Despite this limitation the use of a mean
rank and a range of ranks was deemed to be a useful way to provide a semi-quantitative
comparison of the overall significance of inputs during the course of the 30 day simulation
period.
Figure 5-7 (a) shows the mean rank and inter-daily range of ranks for each input based on
PCC. The input abbreviations are given in Table A.1 in Appendix A. On average, the most
sensitive input in Scenario I was probability of washing body (PWB) with a mean rank of 1.8. The
range of ranks for this input was between 1 and 2, and hence, PWB was among the top two
sensitive inputs throughout the entire month. Inputs with secondary sensitivity with respect to
daily exposures were residue decay rate (DR), body weight (WB), amount of pesticide applied
(AM), fraction of chemicals available for transfer (FTR), and the application area (Area). Average
ranks for these inputs within 30 days of model simulation ranged between 2.1 and 4.7. These
inputs typically had wide ranges of ranks.
In Scenario II, FTR was identified as the most sensitive input throughout the entire month,
and hence, no range of ranks was associated with this input. Transfer coefficient via body
(TCBody), exposure duration at target area (EDTarget), and body washing removal efficiency (BW)
53
(a) Pearson Correlation
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
ank
Scenario I Scenario II Scenario III
(b) Spearman Correlation
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PW
B AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
ank
Scenario I Scenario II Scenario III
Figure 5-7. Comparison of Mean Ranks and Range of Ranks Representing Minimum and Maximum Ranks of Inputs in 30 Days of Model Simulation for the Three Temporal Scenarios Based on: (a) Pearson Correlation Coefficients; and (b) Spearman Correlation
Coefficients.
54
were identified as inputs with secondary sensitivity with mean ranks between 3.1 and 5.6. Inputs
that were identified with secondary sensitivity in Scenario I, typically were among the inputs
with low sensitivity with respect to the incremental change in daily exposures. For example, DR
had a mean rank of 8.4 in Scenario II with a range of ranks between 2 and 20. However, DR has
a montly sampling frequency and for reasons previously discussed would not be expected to be
of significant importance in Scenario II.
In Scenario III, the inputs that have the most importance and secondary importance are
approximately the same as those for Scenario I y. The only exception was with respect to
sensitivity of FTR to cumulative daily exposure. FTR had a low sensitivity with a mean rank of
14.2 and a range of ranks between 1 and 24 in Scenario III.
However, as noted in the previous section, if the main interest is with respect to which
inputs are important to cumulative exposure, then the rankings for cumulative exposure at the
end of the overall time period of interest (e.g., one month in this case) may be of more usefulness
than the average daily rank.
Figure 5-7 (b) shows the mean rank and corresponding range of ranks for each input
based on SCC. Typically, SCC provided the same results with respect to inputs with the most
and secondary sensitivity in the three temporal scenarios. However, comparison of results for the
two correlation-based methods shows that typically inputs that had high sensitivity based on the
two methods, on average had lower mean ranks (i.e., higher sensitivity) based on SCC. For
example, amount of pesticide applied on a monthly basis (AM) had mean ranks of 3.3, 9.9, and
2.6 in the three temporal scenarios, respectively, based on PCC. However, the mean ranks for AM
were 2.2, 7.5, 1.5 in the three temporal scenarios, respectively, based on SCC. Moreover, the
range of ranks associated with each input was narrower when Spearman correlation coefficients
55
were used to rank the inputs. For example, AM was ranked between 2 and 24 and between 3 and
16 based on PCC and SCC, respectively, in Scenario II. An implication is that the SCC results
may be more “stable” than the PCC results, and thus more robust. Of course, this can be due to
nonlinearity effects in the model, which are better accounted for using SCC than PCC.
5.2.4 Summary of Results Based on Correlation Coefficients
Summary of key insights and findings based on correlation-based methods include:
• The magnitude of correlation coefficients typically changed for inputs with respect to
time, regardless of whether inputs were sampled on a monthly or daily basis. Thus, there
is clearly a temporal dynamic in the case study. Sensitivity analysis provides insight and
understanding regarding the causes of the temporal dynamic, and regarding the relative
importance of inputs at different times of the month and with respect to different methods
of quantifying exposure over time (e.g., Scenarios I, II, an d III).
• For inputs that were sampled monthly, there were some general trends in the temporal
pattern of sensitivity. Monthly inputs can either increase or decrease in sensitivity over
time with respect to daily exposures. However, they tend to be unimportant, except
perhaps in the first few days, with regard to incremental changes in exposures. This
result is not inconsistent with situations in which an input is important on a daily basis.
For example, a monthly input can persistently contribute to inter-individual variability in
daily exposures from one day to the next, but might not contribute significantly to
marginal changes in exposures for a given individual from one day to the next.
• Monthly inputs tend to increase in sensitivity with respect to cumulative exposures during
the first few days of the month. For those monthly inputs that increase in importance for
daily exposure as time elapses, there will be a monotonic increase in sensitivity for
56
cumulative exposure as well. However, the marginal increase in sensitivity in such cases
diminishes, at least in the case studies here, because the daily exposures decrease in
magnitude with time. Therefore, there is a dimishing contribution to the cumulative
exposure.
• Correlation coefficients for inputs with a daily sampling strategy typically declined
monotonically with respect to time for both daily and cumulative exposures. A key
reason for this is because of the independence assumption assumed in the case study,
leading to random fluctuations in daily values of such inputs for each individual. For a
linear model in particular, the fluctuations tend to “average out,” with the net effect being
that the influence of the input is really based on the sampling distribution of its mean,
rather than on variability in individual sample values.
• Although inputs sampled with a daily frequency typically become unimportant in
Scenarios I and III by late in the month, they can be important for Scenario II throughout
the month. This is because such inputs can significantly influence day-to-day changes in
the magnitude of daily exposures, even though they may not significantly influence inter-
individual variability in exposures because of the averaging effect discussed above.
• When comparing SCC to PCC, the rank transformation typically led to correlation
coefficients of equal or greater magnitude. For very small correlations, the two methods
provide approximately the same numerical result. For larger correlations (e.g., above
approximately 0.2), the two methods diverge in results. For correlations with a large
magnitude, the SCC coefficients were often statistically significantly greater than those
from PCC. The numerical values at which the two coefficient estimates diverge may be
in part an artifact of the specific model and input assumptions used here. However, in
57
general, for a model that deviates in some way from pure linearity, but that is monotonic,
the SCC is expected to provide a larger numerical value than the PCC for strong
dependencies.
• An implication of the previous finding is that both PCC and SCC may be equally capable
of identifying inputs that are little or no significance. This could be important, for
example, if these methods are used as a screening step prior to application of a more
refined, but also more computationally intensive, sensitivity analysis method. For
example, if many inputs are found to be insignificant or unimportant, their variation
could either be ignored or they could be set to point estimates, and then more
computationally intensive but more accurate sensitivity analysis methods can be applied
to quantify and discriminate the importance among those inputs of high or secondary
importance.
• The SCC appears to be a more robust measure of sensitivity than the PCC, in that there is
typically less variability in the ranks that are inferred by comparing SCCs for all inputs
on a given day than there is based on the PCCs. Thus, the use of SCC appears to produce
more consistent findings regarding the relative importance of each input. As such, SCC
would typically be preferred over PCC.
In general, the results based on correlation coefficients of both types were approximately
comparable, but preference should be given to SCC over PCC at least with regard to applicability
and robustness for the type of modeling used as a testbed here.
In the following sections, the results of additional methods are introduced. This section
provided detailed interpretation and explanation of the key trends and findings from a time-based
58
sensitivity analysis approach. These details will not be repeated in the following sections.
Instead, focus will be given to any key differences in interpretation or insights.
5.3 Regression Analysis
This section presents the results based upon the use of standardized sample (SRA) and
rank linear regression analyses (RRA) to characterize the sensitivity of the selected model
outputs for temporal Scenarios I, II, and III to each of the inputs of the model. Linear effects of
the inputs listed in Table A.1 in Appendix A are included in the regression model. For simplicity,
no interaction effects between inputs were considered. Selected inputs that are the same as those
presented in Section 5.2 are considered as the focus of discussion here and specific details are
given.
This section begins with a discussion of the diagnostic checks that can be performed on
regression results prior to interpreting the sensitivity results. In particularly, changes in the
coefficient of determination, R2, of the regression model with respect to time are given (Section
5.3.1). Regression coefficients are given and compared for the selected inputs (Section 5.3.2).
Changes in ranking of selected inputs with respect to time are presented (Section 5.3.3). Mean
ranks and associated range of ranks in 30 days of model simulation for all inputs are also
provided and compared (Section 5.3.4). A summary of the key findings based on the two
regression-based methods are discussed (Section 5.3.5).
5.3.1 Coefficient of Determination
Coefficient of determination, R2, is used to judge the adequacy of the regression model
based on the linear effects of inputs at each time step of the model simulation. R2 represents the
amount of output variability explained or accounted for by the regression model. At each time
step, the coefficient of determination, R2, was estimated for regression models based on both
59
sample and rank transformed data values. Evaluation of the R2 values provided a diagnostic
check on the results. Generally, a high R2 value implies that the regression model captured a
substantial amount of output variability, and hence, the sensitivity analysis results based on the
relative magnitude of regression coefficients deemed to be reliable.
Figure 5-8 shows the variation in R2 values with respect to time. R2 values are shown for
each of the three temporal scenarios. Except for Scenario III, the amount of output variability
that was captured by the regression model based on the sample data (i.e. the sample regression
method) decreased with time. However, rank transformation of the data typically improved the
output variability captured by the regression model. For example, approximately 60 percent of
the output variability was captured by the regression model based on the sample data in the first
10 days in Scenario I, while this amount ranged between low values of 35 and 23 percent in the
last 10 days of the model simulation. Rank transforming data substantially improved the R2
values within the latter time period. The R2 values based on the ranked transformed data ranged
between high values of 0.88 and 0.92 within the last 10 days of the model simulation. R2 values
of 0.9 or higher are considered to be excellent, implying that the regression model used as a basis
for estimating sensitivity is capturing the vast majority of variability in the exposure model, and
thus it is not likely that any significant source of variability in the exposure model results would
be unaccounted for. In contrast, an R2 value of 0.3, such as occurs for sample regression in
60
(a) Scenario I
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
R2 V
alue
Sample Regression Rank Regression
(b) Scenario II
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
R2 V
alue
Sample Regression Rank Regression
(c) Scenario III
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
R2 V
alue
Sample Regression Rank Regression
Figure 5-8. Variation in Coefficient of Determination, R2, with Respect to Time based on Sample and Rank Regression Analyses for: (a) Scenario I: Daily Exposure; (b) Scenario II:
Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Exposure.
61
Scenario I late in the month, would imply that most of the variation in the exposure model is not
accounted for in the sensitivity analysis. Thus, one might have little confidence that the
sensitivity indices, or the resulting rank order of inputs, would be reliable. Of course, in practice,
the two methods may agree for many kinds of models.
As will be shown later in this section, both sample and rank regression actually provide
similar insights regarding sensitivity. However, as a general rule, and particularly if only one of
these two methods will be applied, it would be prudent to give preference to the method that
more robustly produces reliable results.
For Scenario III, typically R2 values based on sampled and rank transformed data did not
change in time. Moreover, rank transforming data did not improve the amount of output
variability captured by the regression model substantially. The similarity of the two results
implies that the cumulative exposure estimate is more linear than the daily exposure estimate.
This is likely because a cumulative exposure is inherently additive, based upon summing the
daily exposures. Thus, even though there maybe nonlinear effects between daily exposure and a
given input, these nonlinearities may tend to “average out” when adding exposures over multiple
days. Furthermore, even though the diagnostic check for the daily exposures implies significant
divergence between sample and rank regression late in the month, the magnitude of daily
exposures late in the month is small. Therefore, the effect of nonlinearities in daily exposure late
in the month has minor or little effect on cumulative exposure during that time.
5.3.2 Regression Coefficients
For each of the three temporal scenarios, regression coefficients were estimated at each
day for all inputs, and hence, 30 sets of standardized sample regression coefficients and rank
regression coefficients were estimated. Typically, regression coefficients based on both sampled
62
and rank transformed data varied in time in the three temporal scenarios. Similar to correlation-
based methods, for inputs with a monthly sampling strategy, two distinct patterns with respect to
the change in the absolute values of regression coefficients were identified for Scenario I: (1)
monotonic increase; and (2) monotonic decrease. In contrast, inputs with a daily sampling
strategy typically had coefficients with decreasing trend with respect to time for Scenario I.
Examples are provided to illustrate these patterns.
Figure 5-9 shows variation in absolute values of regression coefficients based on SRA
and RRA and corresponding 95% confidence intervals for the residue decay rate (DR) with
respect to time. The coefficients are shown for the three temporal scenarios. DR has a monthly
sampling strategy and provides an example in which the input has relatively higher regression
coefficient later in the month. Overall, the trends here, and the explanations for the tends, are the
same as for the correlation coefficients for each of the three scenarios.
Figure 5-9 (a) shows the results for Scenario I. DR has a substantially higher regression
coefficient later in the month. For example, regression coefficients based on SRA and RRA had
low values of 0.04 and 0.06, respectively, on Day 1. However, there was substantial increase in
the magnitude of these coefficients by the end of the month. The regression coefficient for DR
reached high values of 0.4 and 0.9 based on SRA and RRA, respectively, by the end of the
month.
Figure 5-9 (b) shows the results for Scenario II for the change in regression coefficients
of DR based on SRA and RRA. These results are qualitative similar to those for correlation
coefficients, including the trends over time and that the rank-based method provided larger
magnitudes of the sensitivity index later in the month than did the sample-based method.
63
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
Figure 5-9. Absolute Values and Corresponding 95% Confidence Intervals of Sample and Rank Regression Coefficients for Residue Decay Rate as an Input with a Monthly Sampling Strategy
for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
95% Confidence Interval of the Coefficient
64
Figure 5-9 (c) shows the results for Scenario III. Absolute values of regression
coefficients for DR are shown for SRA and RRA along with their 95% confidence intervals in 30
days of model simulation. The temporal trend and the comparison between rank and sample
based methods are similar to those for PCC and SCC as explained in the previous section.
Comparing regression coefficients based on SRA and RRA for the three temporal
scenarios indicates that rank transforming typically increased the magnitude of the coefficients
for DR. However, this increase in the magnitude of the regression coefficients is more
pronounced with respect to Scenario I. Considering high values of R2 for the regression model
based on the rank transformed data in Scenario I (Figure 5-8), regression coefficients based on
RRA provides a much more reliable measure of the sensitivity of daily exposures to DR than
based on SRA.
For Scenario II, rank transforming the data did not improve the magnitude of regression
coefficients. Because R2 values for the regression model in this scenario were typically low, it
matters less as to what type of regression analyses is used.
In Scenario III, rank transforming the data had a minimal effect on improving the
magnitude of regression coefficients, similar to its effect on R2 values for this scenario.
Confidence intervals for the regression coefficients provide insight with respect to
statistically significant differences between coefficients. Typically, regression coefficient based
on SRA and RRA were comparable in the first few days of the model simulation due to
overlapping confidence intervals. However, later in the month, typically these two regression-
based methods provided coefficients which were statistically significantly different for all three
Scenarios. However, for Scenario III, the magnitude of the differences were less pronounced,
even though they are statistically significant.
65
Significant changes in regression coefficients from one time period to another can also be
assessed by confidence intervals for each of the regression-based methods. For example, for
Scenario I, regression coefficients based on the rank transformed data were statistically different
within the first 15 days of the model simulation. However, through the rest of the month, these
coefficients were typically comparable for time periods of several adjacent days.
Figure 5-10 shows variation in absolute values of regression coefficients based on SRA
and RRA and corresponding 95% confidence intervals for probability of washing body (PWB)
with respect to time. PWB has a monthly sampling strategy and provides an example in which the
highest regression coefficient for Scenario I occurs early in the month. Overall, the results and
key findings are approximately similar to those obtained from PCC and SCC, including the
temporal trends in each scenario and the comparison of rank versus sample-based methods.
In Scenario I, the magnitude of regression coefficients is relatively modest, ranging from
approximately 0.1 to 0.2. However, the coefficients are significantly larger within the first 10
days when compared to later in the month. The reason for the peak in the sensitive indices near
the 5th day is explained in the section on PCC and SCC. For Scenario II, the regression
coefficients decline from the first day to about the 5th to 10th day, and there after remain
approximately constant at values that are either practically or statistically insignificant.
The magnitude of regression coefficients increases with time with respect to cumulative
exposure in Scenario III, as shown in Figure 5-10 (c), during the first five days of the month.
However, through the rest of the month, there is no significant change in regression coefficients
based on either method, for reasons explained previously.
For the two regression-based methods, typically regression coefficients were
comparable in different time steps of the model simulation. Confidence intervals for the
66
(a) Scenario I: Daily Exposure
0
0.1
0.2
0.3
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(b) Scenario II: Incremental Change in Daily Expsoure
0
0.1
0.2
0.3
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(c) Scenario III: Cumulative Exposure
0
0.1
0.2
0.3
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
Figure 5-10. Absolute Values and Corresponding 95% Confidence Intervals of Sample and Rank Regression Coefficients for Probability of Washing Body as an Input with a Monthly
Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
95% Confidence Interval of the Coefficient
67
regression coefficients based on the two methods typically overlapped. In general, the regression
coefficients were relatively small for all scenarios and time steps. Thus, it appears to be the case
that for an input that has a small sensitivity index, the two methods provide comparable results.
Typically, the regression coefficients based on SRA and RRA did not statistically differ
from one time period to another in the second half of the month for the three temporal scenarios.
Compared to DR, the main difference is that PWB decreases in sensitivity on a daily basis, where
as DR increased in sensitivity on a daily basis, as explained in more detail in Section 5.2.
Figure 5-11 illustrates an example of change in regression coefficients for the fraction of
chemicals available for transfer (FTR). This input has a daily sampling strategy. When compared
to the results from correlation coefficients as shown in Figure 5-4, the general trends are
approximately the same. However, the regression results appear to be more consistent, with less
noisy perturbations, and the sensitivity measures appear to have narrower confidence intervals on
a relative basis. Thus, the sensitivity indices based on regression coefficients appear to be
perhaps more stable and more robust than those based on correlation coefficients.
Except for the first few days in Scenarios I and III, regression coefficients based on SRA
and RRA were statistically comparable. In Scenario II, rank transforming the data significantly
increased the magnitude of the regression coefficients. This is similar to the findings in Section
5.2.
5.3.3 Rankings for Selected Inputs
At each time step, inputs were ranked based on the relative magnitude of the absolute
regression coefficients. The rank of one is assigned to an input with highest regression
coefficient. A similar pattern compared to rankings based on correlation-based methods is
observed for the results from SRA and RRA. Some inputs had relatively higher sensitivity in the
68
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
|Reg
ress
ion
Coe
ffici
ent|
Sample Regression Rank Regression
Figure 5-11. Absolute Values and Corresponding 95% Confidence Intervals of Sample and
Rank Regression Coefficients for Fraction of Chemicals Available for Transfer as an Input with a Daily Sampling Strategy for: (a) Scenario I: Daily Exposure; (b) Scenario II: Incremental
Change in Daily Exposure; and (c) Scenario III: Cumulative Daily Exposure.
95% Confidence Interval of the Coefficient
69
(a) Sample Regression
0
5
1015
20
25
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Rank Regression
05
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-12. Variation in Rank of Residue Decay Rate as an Input with Monthly Sampling Strategy with Respect to Time Based on: (a) Sample Regression Analysis; and (b) Rank
Regression Analysis.
earlier time steps, while other inputs showed higher sensitivity in later time steps of the model
simulation. Typically, inputs with monthly sampling frequencies had higher sensitivity in later
time steps, while inputs with daily sampling frequencies were among the list of inputs with top
sensitivity in earlier time steps of the model simulation. Residue decay rate (DR) and the fraction
of chemicals available for transfer (FTR) are selected as two examples in order to illustrate the
pattern of change in sensitivity of an input on a daily basis.
Figures 5-12 shows variation in ranks of DR with respect to time for the three temporal
scenarios. The ranks are based on SRA and RRA in Figures 5-12 (a) and 5-12 (b), respectively.
DR typically showed lower sensitivity earlier in the month, while its sensitivity increased later in
the month for all three temporal scenarios. For example, for Scenario II, while DR had a range of
Lower Rank Implies Greater Sensitivity
70
(a) Sample Regression
05
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Rank Regression
0
5
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-13. Variation in Rank of Fraction of Chemical Available for Transfer as an Input with
a Daily Sampling Strategy with Respect to Time Based on: (a) Sample Regression Analysis; and (b) Rank Regression Analysis.
ranks between 5 and 24 in the first 16 days of the model simulation based on SRA, it was
identified as the second most sensitive input through out the rest of the month.
Figure 5-13 shows an example in which selected input had relatively higher sensitivity in the
earlier time steps of the model simulation. FTR has a daily sampling frequency and typically had
higher sensitivity within the first 5 days of the model simulation compared to the rest of the
month for daily and cumulative exposure scenarios. However, FTR was identified as the most
important input throughout the entire month in Scenario II. Although the qualitative trends for
the rankings over time are similar to those obtained from the correlation methods, the
quantitative results appear to have some significant differences. For example, in Figure 5-6, the
rankings for Scenario III imply little or no importance in the second half of the month. However,
in Figure 5-13, the rankings for Scenario III imply moderate importance, with ranks typically of
approximately seven. Furthermore, there is less inter-daily variability in the rankings for
Lower Rank Implies Greater Sensitivity
71
Scenario III in the second half of the month. This implies that the regression method is
responding differently than the correlation methods. However, the R2 values for Scenario III are
typically around 0.6 to 0.8. This implies that the regression model captures or explains most of
the variability in the exposure model, but that a substantial portion of variability is not addressed.
Thus, while the results from the regression methods may appear to be more stable, it is not clear
that they are any more reliable than the correlation methods. In either case, the sample-based
methods presume a linear response and the rank-based methods presume a monotonic response.
Thus, the major theoretical underpinnings with respect to sensitivity analysis are approximately
the same
5.2.4 Comparison of Mean Ranks
This section presents the mean rank associated with each input within the time period of the
model simulation. The mean rank represents the arithmetic average of ranks for each input. The
range of ranks also is given for each input. The range of ranks represents minimum and
maximum ranks for the input within 30 days of model simulation. In general qualitative terms,
the results for the mean ranks shown in Figure 5-14 based on the regression methods are
comparable to those shown in Figure 5-7 for the correlation methods. However, there are some
quantitative differences.
Figure 5-14 (a) shows the mean rank and corresponding range of ranks for each
input based on SRA. The input abbreviations are given in Table A.1 in Appendix A. On average,
the most sensitive input in Scenario I was fraction of chemicals for transfer (FTR) with a mean
rank of 2.0. The range of ranks for this input was between 1 and 4. Inputs with secondary
sensitivity with respect to daily exposures were residue decay rate (DR), amount of pesticides
applied (AM), area of pesticide application (Area), body weight (WB), and probability of
72
(a) Sample Regression Analysis
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Input
Mea
n R
ank
Scenario I Scenario II Scenario III
(b) Rank Regression Analysis
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PWB
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
EDno
n-ta
rget
FTR
FBU HF
Input
Mea
n R
ank
Scenario I Scenario II Scenario III
Figure 5-14. Comparison of Mean Ranks and Range of Ranks Representing Minimum and Maximum Ranks of Inputs in 30 Days of Model Simulation for the Three Temporal Scenarios Based on: (a) Sample Regression Analysis; and (b) Rank Regression Analysis.
73
washing body (PWB). Average ranks for these inputs within 30 days of model simulation ranged
between 2.3 and 5.5. These inputs typically had wide ranges of ranks.
In Scenario II, FTR was identified as the most sensitive input throughout the entire month,
and hence, no range of ranks was associated with this input. Transfer coefficient via body
(TCBody), exposure duration at target area (EDTarget), and body washing removal efficiency (BW)
were identified as inputs with secondary sensitivity with mean ranks between 4.7 and 5.4. Inputs
that were identified with secondary sensitivity in Scenario I typically were among the inputs with
low sensitivity with respect to the incremental change in daily exposures. For example, DR had a
mean rank of 8.0 in Scenario II with a range of ranks between 2 and 24.In Scenario III, AM was
identified as the most sensitive input with a mean rank of 2.0 in 30 days of model simulation.
Inputs with secondary sensitivity included WB, DR, Area, and FTR with mean ranks between 2.7
and 5.2.
Figure 5-14 (b) shows the mean rank and corresponding range of ranks for each input
based on RRA. Typically, RRA provided the same results with respect to inputs with the most
and secondary sensitivity in the three temporal scenarios. However, comparison of results for the
two regression-based methods shows that typically inputs that had high sensitivity based on the
two methods, on average had lower mean ranks (i.e., higher sensitivity) based on RRA.
Moreover, the range of ranks associated with each input was narrower when rank regression
coefficients were used to rank the inputs. This is similar to the findings based on comparison of
SCC to PCC.
When comparing the regression-based results to the correlation-based results, there are
some perhaps minor differences in the rankings. For example, PBW tends to have less sensitivity
according to the regression methods, whereas FTR tends to have slightly more sensitivity. For
74
FTR, all of the methods imply more or less the same overall level of importance. However, for
PBW, the regression methods imply a significantly lower ranking especially for Scenarios I and
III. Because of the very high R2 for RRA in Scenario I, the findings based on this method might
be treated with some deference. However, there does not appear to be much different in average
ranking even for this input when comparing SRA to RRA, despite the much lower R2 for the
former.
5.2.5 Summary of Results Based on Regression-Based Methods
The key insights and findings based on regression-based methods include:
• Similar to the findings with PCC and SCC, the magnitude of regression coefficients
typically changed for inputs with respect to time, regardless of whether inputs were
sampled on a monthly or daily basis. Thus, there is clearly a temporal dynamic in the
case study.
• For inputs that were sampled monthly, there were some general trends in the temporal
pattern of sensitivity. Monthly inputs can either increase or decrease in sensitivity
over time with respect to daily exposures. However, they tend to be unimportant,
except perhaps in the first few days, with regard to incremental changes in exposures.
• Monthly inputs tend to increase in sensitivity with respect to cumulative exposures
during the first few days of the month. For those monthly inputs that increase in
importance for daily exposure as time elapses, there will be a monotonic increase in
sensitivity for cumulative exposure as well. However, the marginal increase in
sensitivity in such cases diminishes, at least in the case studies here, because the daily
exposures decrease in magnitude with time. Therefore, there is a diminishing
contribution to the cumulative exposure.
75
• Regression coefficients for inputs with a daily sampling strategy typically declined
monotonically with respect to time for both daily and cumulative exposures. A key
reason for this is because of the independence assumption assumed in the case study,
leading to random fluctuations in daily values of such inputs for each individual. For
a linear model in particular, the fluctuations tend to “average out,” with the net effect
being that the influence of the input is really based on the sampling distribution of its
mean, rather than on variability in individual sample values.
• Although inputs sampled with a daily frequency typically become unimportant in
Scenarios I and III by late in the month, they can be important for Scenario II
throughout the month. This is because such inputs can significantly influence day-to-
day changes in the magnitude of daily exposures, even though they may not
significantly influence inter-individual variability in exposures because of the
averaging effect discussed above.
• When comparing RRA to SRA, the rank transformation typically led to regression
coefficients of equal or greater magnitude. However, in general, for a model that
deviates in some way from pure linearity, but that is monotonic, the RRA method is
expected to provide a more robust numerical value than the SRA method.
• An implication of the previous finding is that both SRA and RRA may be equally
capable of identifying inputs that are little or no significance. This could be
important, for example, if these methods are used as a screening step prior to
application of a more refined, but also more computationally intensive, sensitivity
analysis method. For example, if many inputs are found to be insignificant or
unimportant, their variation could either be ignored or they could be set to point
76
estimates, and then more computationally intensive but more accurate sensitivity
analysis methods can be applied to quantify and discriminate the importance among
those inputs of high or secondary importance.
In general, the results and findings from the regression methods are similar to those from the
correlation methods. However, there are some differences in quantitative results.
5.4 Analysis of Variance
This section presents the results based upon the use of ANOVA to characterize the
sensitivity of the selected model outputs for temporal Scenarios I, II, and III to each of the inputs
of the model. The main (direct) effects of the inputs listed in Table A.1 in Appendix A are
included in the ANOVA analysis. For simplicity, no interaction effects between inputs were
considered. However, such effects could be considered by expanding the scope of application of
ANOVA to explicitly include interaction terms between selected combinations of inputs.
Inputs in the simplified SHEDS-Pesticides model were continuous, and hence, must be
partitioned into levels prior to application of ANOVA (Kleijnen and Helton, 1999). Frey et al.
(2004) demonstrated three approaches for defining levels for continuous inputs based on: (1)
evenly spaced intervals; (2) evenly spaced percentiles; and (3) visual inspection of the
cumulative distribution function (CDF) for each input. For simplicity and in order to be
consistent with respect to defining levels at different time steps of the model simulation, the
evenly spaced percentile approach was used to define levels. In this approach the CDF of the
generated values for an input in a probabilistic simulation is used for defining levels at evenly
spaced percentiles. For each input, four levels were defined based on the 25th, 50th, and 75th
percentiles of the generated samples at each time step.
77
F values are used to measure sensitivity and are given and compared for the selected
inputs (Section 5.4.1). Changes in ranking of selected inputs with respect to time are presented
(Section 5.4.2). Mean ranks and associated range of ranks in 30 days of model simulation for all
inputs are also provided and compared (Section 5.4.3). A summary of the key findings based on
ANOVA are discussed (Section 5.4.4).
5.4.1 F Values
For each of the three temporal scenarios, F values associated with inputs in the ANOVA
model were estimated at each day, and hence, 30 sets of F values were estimated. Typically, F
values varied in time in the three temporal scenarios. The F values had similar patterns with
respect to change in values in time compared to correlation and regression-based methods
discussed in previous sections. For the purpose of illustration, two examples are given.
Figure 5-15 (a) shows variation in F values of residue decay rate (DR). Similar to the
pattern shown for correlation and regression coefficients, F values for DR typically
monotonically increased in time for the three temporal scenarios. However, the magnitude of F
values was substantially higher in Scenario I compared to the other two scenarios. F Values for
DR reached a high value of approximately 350 on Day 10 and maintained approximately the
same high value through the rest of the month. The trends for Scenarios II and III imply the
same general findings as were obtained from either correlation or regression methods.
Figure 5-15 (b) shows variation in F values for fraction of chemicals available for transfer
(FTR) with respect to time. In contrast to DR, F values corresponding to FTR were larger in earlier
time steps of the model simulation, and declined later in the month similar to the pattern shown
for correlation and regression coefficients in Sections 5.2 and 5.3. Because rankings are based on
the relative magnitude of F values at each time step, these inputs are expected to have different
78
(a) Residue Decay Rate
0
50
100
150
200
250
300
350
400
0 5 10 15 20 25 30
Time (day)
F Va
lue
Scenario I Scenario II Scenario III
(b) Fraction of Chemical Available for Transfer
0
50
100
150
200
250
300
350
0 5 10 15 20 25 30
Time (day)
F Va
lue
Scenario I Scenario II Scenario III
Figure 5-15. Variation of F Values with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer.
sensitivity throughout the month. The rankings of these two inputs are presented in the next
section.
5.4.2 Rankings for Selected Inputs
At each time step, inputs were ranked based on the relative magnitude of the F values. A
rank of one is assigned to an input with highest F value. The change in ranks for input with
respect to time followed a similar pattern compared to correlation and regression-based methods.
79
(a) Residue Decay Rate
05
10152025
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
(b) Fraction of Chemicals Available for Transfer
0
5
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-16. Variation of Ranks with Respect to Time for the Three Temporal Scenarios for: (a)
Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer.
Some inputs had relatively higher sensitivity in the earlier time steps, while other inputs
showed higher sensitivity in later time steps of the model simulation. Typically, inputs with
monthly sampling frequencies had higher sensitivity in later time steps, while inputs with daily
sampling frequencies were among the list of inputs with top sensitivity in earlier time steps of the
model simulation, particularly for Scenarios I and III. Inputs that were sampled on a daily basis
typically had more influence on the results of Scenario II, particularly later in the month.
Figure 5-16 (a) shows variation in ranks of DR with respect to time for the three temporal
scenarios. DR typically showed lower sensitivity earlier in the month, while its sensitivity
increased later in the month for the three temporal scenarios. For example, while DR had a range
of ranks between 3 and 22 in the first 15 days of the model simulation, it was typically identified
Lower Rank Implies Greater Sensitivity
80
as the second most sensitive input through out the rest of the month in Scenario II. The results
here are very similar to those from correlation and regression methods for all three scenarios.
Figure 5-16 (b) shows an example in which the selected input had relatively higher
sensitivity in the earlier time steps of the model simulation. FTR has a daily sampling strategy and
typically had higher sensitivity within the first 5 days of the model simulation compared
to the rest of the month for daily and cumulative exposure scenarios. However, FTR was
identified as the most sensitive input throughout the entire month in Scenario II. The results
from ANOVA in Figure 5-16 for FTR are quantitatively more similar to those from the
correlation methods in Figure 5-6 than for the regression methods as shown in Figure 5-13. In
particular, the rankings for Scenario III late in the month are very similar for ANOVA and the
correlation methods, with what appears to be more fluctuation in ranks on a daily basis. As
discussed in Section 5.2, these rankings simply imply that there is a group of inputs all of which
are either of minor or insignificant importance, and thus changes in rankings within such a group
of inputs is not of any practical significance.
5.4.3 Comparison of Mean Ranks
This section presents the mean rank associated with each input within the time period of
the model simulation. The mean rank represents the arithmetic average of ranks for each input.
The range of ranks also is given for each input. The range of ranks represents minimum and
maximum ranks for the input within 30 days of model simulation.
Figure 5-17 shows the mean rank and corresponding range of ranks for each input based
on ANOVA. The input abbreviations are given in Table A.1 in Appendix A. On average, the
most sensitive input in Scenario I was residue decay rate (DR) with a mean rank of 1.9. The range
of ranks for this input was between 1 and 10. Inputs with secondary sensitivity with respect to
81
0.0
5.0
10.0
15.0
20.0
25.0
Cai
r
Cb
WB k
PH
PWB
AM
Are
a
RN
/T
Hei
ght
DR
PAI
Din
h
HW
BW LH LB Ef
TCha
nd
TCbo
dy
EDta
rget
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
anks
Scenario I Scenario II Scenario III
Figure 5-17. Comparison of Mean Ranks and Range of Ranks of Inputs in 30 Days for the Three
Temporal Scenarios Based on ANOVA.
daily exposures were probability of washing body (PWB), amount of pesticides applied (AM),
area of pesticide application (Area), fraction of chemicals available for transfer (FTR), and body
weight (WB). Average ranks for these inputs within 30 days of model simulation were between
2.1 and 5.7.These inputs typically had wide ranges of ranks.
In Scenario II, FTR was identified as the most sensitive input throughout the entire month,
and hence, no range of ranks was associated with this input. Transfer coefficient via body
(TCBody), exposure duration at target area (EDTarget), and body washing removal efficiency (BW)
were identified as inputs with secondary sensitivity with mean ranks between 4.3 and 5.4. Inputs
that were identified with secondary sensitivity in Scenario I, typically were among the inputs
with low sensitivity with respect to the incremental change in daily exposures. For example, DR
had a mean rank of 6.5 in Scenario II with a range of ranks between 2 and 22.
In Scenario III, PWB was identified as the most sensitive input with a mean rank of 1.5 in
30 days of model simulation. Inputs with secondary sensitivity included AM, DR, Area, and WB
with mean ranks between 2.6 and 4.8.
Overall, the average rankings obtained from ANOVA are similar to those for the
correlation methods. The rankings from ANOVA differ quantitatively in some ways from those
82
obtained from the regression methods, in the same manner in which the correlation-based results
differ somewhat from the regression-based results. This implies that ANOVA is providing
insights perhaps more similar to those from the correlation-based than the regression-based
methods. However, in a general sense, all three sets of methods are providing comparable
insights.
Although not explored here quantitatively, ANOVA and regression have the advantage of
being expanded to deal with selected user-specified interaction effects. In this regard, ANOVA
would be expected to have an advantage over regression methods in that a functional form for
the interaction effect need not be assumed.
5.4.5 Summary of Results Based on ANOVA
The key insights and findings from ANOVA are essentially the same as those obtained
from the correlation methods as detailed in Section 5.2.4. The details are not repeated here.
However, the general findings are briefly summarized:
• The magnitude of F values typically changed for inputs with respect to time.
• F values for inputs with a monthly sampling strategy typically in Scenario I: (1) declined
monotonically; or (2) increased monotonically with respect to time.
• F values for inputs with a daily sampling strategy typically declined monotonically with
respect to time in Scenarios I and III, but often are high for Scenario II.
• Results for Scenario II for the top sensitive inputs were typically different from those in
the other two temporal scenarios.
• ANOVA produces results that are more similar to correlation than regression methods.
However, ANOVA has the advantage of additional flexibility compared to correlation
methods, and of being model-independent when compared to regression methods, in that
83
it is possible to add interaction terms and account for these in determining key
associations with the variation in the model output.
5.5 Sobol’s Method
This section presents results based upon the use of Sobol’s method to characterize the
sensitivity of the selected model outputs for temporal Scenarios I, II, and III to each of the inputs
of the model. Unlike the previous methods, Sobol’s method is intended to provide an
apportionment of the variance in the output with respect to model inputs and interaction effects
among model inputs. In order to estimate the uncertainty in main effect and total effect
sensitivity indices, bootstrap simulation with sample size of 500 is performed. An uncertainty
range of ± one standard error around the mean value is reported for each sensitivity index.
Selected inputs the same as those presented in previous sections are used as the focus of
discussion here and specific details are given.
A discussion is given regarding the contribution of inputs to the output variance (Section
5.5.1). Main and total effect sensitivity indices for selected inputs are given and compared
(Section 5.5.2). Changes in ranking of selected inputs with respect to time are presented (Section
5.5.3). Mean ranks and associated range of ranks in 30 days of model simulation for all inputs
are also provided and compared (Section 5.5.4). The key findings based on Sobol’s method are
discussed (Section 5.3.5).
5.5.1 General Insight Regarding Contribution of Inputs to the Output Variance
For each of the three temporal scenarios, Sobol’s method was applied to the model at
each time step. For each time step, bootstrap simulation was performed to quantify uncertainty
ranges of sensitivity indices including main and total effects. Thus, at each time step 500 main
effect and total effect sensitivity indices were estimated for each input. The uncertainties are
reported in terms of plus or minus one standard error. These are not 95 percent confidence
84
intervals. The reason for reporting in this way is that the bootstrap-p method was used. This
method is considered adequate for estimating standard errors but is not as accurate at estimating
confidence intervals. Either a larger number of bootstrap samples and/or an alternative boostrap
method would be needed for that purpose. However, the range of plus or minus one standard
error is illustrative of the range of uncertainty in the estimates.
The main effect of each input represents the fractional contribution of the input to the
output variance. For a linear model with additive terms, the sum of main effects for all inputs
should equal to one. However, in the case of non-linearity and interactions between inputs, the
sum of main effects will be less than one. Figure 5-18 shows the sum of main effects and
corresponding uncertainty ranges at each time step of the model simulation for each of the three
exposure scenarios.
For Scenario I, as shown in Figure 5-18(a), on average, approximately 60 percent of the
output variance was apportioned to the main effects of inputs within the first 10 days of the
model simulation. However, in the last 10 days of the model simulation, the sum of main effects
relatively decreased to values between 30 and 40 percent. This implies that the model is
increasingly nonlinear later in the monthly simulation period. As discussed in Section 5.2
regarding correlation methods, and as reinforced based on results obtained with regression and
85
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Ave
rage
Sum
of M
ain
Effe
cts
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Ave
rage
Sum
of M
ain
Effe
cts
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Ave
rage
Sum
of M
ain
Effe
cts
Figure 5-18. Variation in Average Sum of Main Effects of Inputs in 500 Bootstrap Simulations and Corresponding Uncertainty Ranges with Respect to Time for the Three Temporal Scenarios.
(Intervals shown represent plus or minus one standard error of the mean value).
86
ANOVA methods, the effect of the residue decay rate becomes more pronounced later in the
month, leading to more nonlinearity in the daily exposure estimates. Thus, it is not surprising
that a declining portion of the variance in the model output for Scenario I is captured by linear
effects. Furthermore, the comparison of rank versus sample-based methods implies that the daily
exposure has a strong monotonic response, but is nonlinear.
Figure 5-18 (b) shows the sum of main effects of inputs at each time step in Scenario II.
In Scenario II, contribution of main effects of inputs to the output variance was substantially
lower in comparison to Scenario I. For example, approximately between 10 and 20 percent of the
output variance was apportioned to the main effects of inputs in the last 10 days of the model
simulation. This finding is qualitatively similar to the insight obtained from analysis of R2
values as shown in Figure 5-8. The model appears to have a significant nonlinear response with
respect to incremental changes from one day to the next, perhaps due to significant interactions
among inputs.
Figure 5-18 (c) shows sum of main effects of inputs at each time step in Scenario III. In
Scenario III, the contribution of main effects of inputs to the output variance did not typically
change at different time steps of the model simulation and was approximately 60 percent. These
findings are consistent with the results shown in Figure 5-8 for the variation of R2 values with
respect to time for a regression model that included linear effects of the inputs.
Typically, at each time step, a few inputs had significantly higher contribution to the
output variance compared to the rest of the inputs. To illustrate this, results are shown in Figure
5-19 for Scenario I for three selected days: Day 1, Day 15, and Day 30. The main effect of the
five inputs that had the largest contribution to variance are shown. Also shown is the sum of the
main effects of all other inputs, and the sum of the interaction effects among all of the inputs.
87
(a) Day 1
BW, 8%
Main Effect of Others,
30%
Interactions, 24%
AM, 6%
WB, 6%
DR, 1%
FTR, 25%
(b) Day 15
Main Effect of Others,
24%
Interactions, 30%
DR, 33%
FTR, 7%
BW, 1%
WB, 2%AM, 3%
(c) Day 30
Interactions, 53%
FTR, 1%
DR, 40%
Main Effect of Others,
3% AM, 1%WB, 1%
BW, 1%
Figure 5-19. Contribution Selected Inputs to the Output Variance in Scenario I for Selected Days: (a) Day 1; (b) Day 15; and (c) Day 30.
FTR: Fraction of Chemicals Available for Transfer DR: Residue Decay Rate WB: Body Weight BW: Body Washing Efficiency AM: Amount of Chemical Applied
88
The inputs that typically were in the top five in terms of greatest contribution to the main
effect include fraction of chemicals available for transfer (FTR), residue decay rate (DR), body
weight (WB), body washing efficiency (BW), and amount of chemical applied (AM). Of course,
because the rankings of these inputs change on a daily basis, on some days some of these inputs
may be outside of the top 5.
On Day 1, FTR accounts for 25 percent of the variance in the model output, which is
nearly three times greater than the next most important input (BW). Thus, there is one input that
is clearly a dominate source of variance in comparison to any other individual input. There are
three inputs of comparable but secondary importance, including BW, WB, and AM, which each
account for 6 to 8 percent of the variance of the output, and combined they account for 20
percent of the variance. The fifth input shown individually, DR, is of tertiary importance,
contributing only one percent to the total variance. The other 20 inputs, combined, contribute
only 30 percent to the variance of the output. However, there are significant interaction effects
among all of the inputs. The interaction effects account for 24 percent of the variance in the
output. If there was a need, it is possible to use Sobol’s method to identify the various
interaction effects more specifically, with a trade-off of increased computational effort. For
example, all of the pairwise interactions could be quantified and compared, as could all of the
three-way interactions, four-way interactions, and so on.
FTR contributes to the output variance not only with its main effect, but also through
interactions with other inputs. The sum of the overall contribution of FTR to the output variance is
given in the total sensitivity index of the input. For example, on Day 1 the total effect of FTR is
0.34. This value is given in Figure 5-21 and discussed further later in Section 5.5.2. Thus, on
Day 1, 34% of the output variance was apportioned to the total effect of FTR. Because the total
89
effect of FTR is the sum of main effect and all possible interaction effects involving FTR, on Day
1, FTR has 9% contribution to the output variance via its interaction with other inputs.
On Day 15, the importance of FTR has waned and DR has emerged as the dominant source
of variability in daily exposure. These two inputs account for 40 percent of the variance. The
other three inputs identified specifically account for only 6 percent of the variance, and thus are
of less importance as a group than FTR is on an individual basis. The other 25 inputs contribute
24 percent of the variance, which is less than on Day 1, but the contribution of interactions has
increased to 30 percent. Thus, there is a clear temporal shift among the inputs and with regard to
interaction effects.
By Day 30, DR clearly is the only individual input that can be clearly distinguished from
any other. DR contributes 40 percent to the variance. The next most important input contributes
only 1 percent to the variance. The 20 inputs not explicitly identified contribute only 3 percent
to the variance. However, the interaction effect has become the single most important source of
variance in the output, accounting for just over half of the total variance. As discussed in earlier
sections, the model behavior is more highly nonlinear at this point in time, and thus it is not
surprising that the interaction effect is so pronounced.
The examples shown in Figure 5-19 illustrate the key benefit of Sobol’s method, which is
to provide a relatively clear and intuitive measure of sensitivity. However, when the interaction
effect becomes large, then additional information may be desirable, which could be obtained at
the cost of a much greater computational burden.
5.5.2 Sensitivity Indices
For each of the three temporal scenarios, main effect and total effect sensitivity indices
based were estimated at each day for all inputs. The mean sensitivity index and standard error
90
associated with each index were estimated based on the results of bootstrap at each day. Thus, 30
sets of mean main effects and mean total effects were estimated at each time step of the model
simulation. Typically, sensitivity indices based on the Sobol’s method varied in time in the three
temporal scenarios. The apportionment to variance shown in Figure 5-19 was obtained by
dividing the individual main effect of each input by the sum of the total effects of all inputs.
This section will focus on use of the raw sensitivity index in order to compare total and main
effects for individual inputs and to evaluate temporal trends.
Figure 5-20 shows variation in average main and total effects and corresponding
uncertainty ranges (based on plus or minus one standard error) with respect to time for the
residue decay rate (DR), which has a monthly sampling frequency. In general terms, the results
are qualitatively similar to those from correlation, regression, and ANOVA methods. For
example, as identified based on results from other methods, as time elapses DR becomes more
important later in the month with respect to variability in daily exposures, the model response
becomes more nonlinear, and the magnitude of daily exposures declines. As shown in Figure 5-
20(a), the main effect increases especially in the first 15 days, and then appears to increase only
slowly, if at all, later in the month. However, the total effect, which includes interactions (e.g.,
nonlinearities) continues to increase substantially over the course of the entire month, which
implies increasing nonlinearity that can be attributed to the nonlinear decay term in the model.
The comparison of the main and total effect from Sobol’s method in relation to the rank
versus sample results for correlation and regression methods suggests that the latter may be
offering insight somewhat similar to Sobol’s method. For example, the rank-based methods
would better account for nonlinearities, whereas the sample-based methods focus on the linear
effect. Thus, the rank-based results tend to produce higher coefficients perhaps because they are
91
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
Sens
itivi
ty In
dex
Main EffectTotal Effect
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
Figure 5-20. Average Main and Total Effects and Corresponding Uncertainty Ranges Based on
500 Bootstrap Simulations for Residue Decay Rate for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure.
92
(a) Scenario I: Daily Exposure
0.00
0.20
0.40
0.60
0.80
1.00
0 5 10 15 20 25 30
Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(b) Scenario II: Incremental Change in Daily Exposure
0.00
0.20
0.40
0.60
0.80
1.00
0 5 10 15 20 25 30
Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(c) Scenario III: Cumulative Exposure
0.00
0.20
0.40
0.60
0.80
1.00
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
Figure 5-21. Average Main and Total Effects and Corresponding Uncertainty Ranges Based on
500 Bootstrap Simulations for Residue Decay Rate for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure.
.
93
accounting for both the linear effect and at least some portion of the nonlinear effect. However,
the correlation methods cannot explicitly account for interactions, and the functional form of the
regression models used here did not include interaction terms. Thus, the correlation and
regression case studies illustrate a distinction perhaps between nonlinear and linear effects
attributable to individual inputs, but do not account for more general forms of interaction as does
Sobol’s method.
Figure 5-21 shows results for the fraction of chemical available for transfer. In general,
these results are qualitatively similar to those from other methods. For example, the sensitivity
of daily exposure to FTR decreases with time, the sensitivity of incremental changes in daily
exposure remains approximately constant at a relatively high sensitivity index, and the sensitivity
of cumulative exposure is relatively low and decreases with time. For the latter, the results by
the end of the month are approximately insignificant, as is the case based on other methods. The
large difference in the total and main effect for Scenario II implies a significant interaction or
nonlinear effect. For the correlation and regression methods, a significant difference was
observed between the rank and sample based methods, which also implies that there is some type
of nonlinear effect. Thus, the comparison of results among these methods is qualitatively
consistent.
5.5.3 Ranking for Selected Inputs
At each time step, inputs were ranked based on the relative magnitude of mean total
effects. Each mean value represents the arithmetic average of 500 bootstrap estimates for the
sensitivity index. The rank of one is assigned to an input with highest mean total effect. The total
effects of inputs accounts for all possible interactions, but additional information would be
needed in order to more specifically determine which specific interactions are contribution the
94
(a) Residue Decay Rate
0
5
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Fraction of Chemicals Available for Transfer
0
5
10
15
20
25
1 5 10 15 20 25 30Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-22. Variation of Ranks with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer Based on the Sobol’s
Method.
most to the total interaction effect. The total effects are more reliable than the main effects in
order to investigate the overall effect of each single input on the output. Typically, a similar
pattern with respect to change in ranks of inputs compared to correlation and regression-based
methods was observed.
Figure 5-22 (a) shows variation in ranks of DR with respect to time for the three temporal
scenarios. DR typically showed lower sensitivity earlier in the month, while its sensitivity
increased later in the month for the three temporal scenarios. For example, while DR had a range
of ranks between 2 and 16 in the first 15 days of the model simulation, it was typically identified
among the top two sensitive inputs through out the rest of the month in Scenario II. Thus, the
Lower Rank Implies Greater Sensitivity
95
key similarities in these results versus those from correlation, regression, and ANOVA methods
are the typical increase in importance for Scenarios I and III as time elapses. However, for
Scenario II, there is some qualitative difference in results. For the other three sets of methods,
there was typically a peak in the sensitivity index at approximately the 5th to 10th day. However,
this peak is much less evident based on the results from Sobol’s method. The overall insight that
DR is important for Scenario II only late in the month is consistent based on results from all of
the methods.
For FTR, the results from Sobol’s method imply that early in the month this input is
important for all scenarios, that this input tends to remain important for Scenario II throughout
the month, and that this input tends to decrease in importance for Scenarios I and II at about the
6th or 7th day. This result is most similar to that obtained based on regression analysis, although
in regression analysis the importance of this input with respect to Scenario I appears to be higher.
The correlation and ANOVA results also imply that early in the month this input decreases in
importance for Scenario I, but also imply that this input is unimportant for Scenario III by
approximately the 10th day. However, it is possible that an input ranked as high as 5th, as implied
by the Sobol’s results for Scenario III late in the month, might contribute very little to the overall
variance in the model output. Thus, a rank of five might not imply any significant magnitude of
importance.
5.5.4 Comparison of Mean Ranks
This section presents the mean rank associated with each input within the time period of
the model simulation. The mean rank represents the arithmetic average of ranks for each input.
The range of ranks also is given for each input. The range of ranks represents minimum and
maximum ranks for the input within 30 days of model simulation. Mean ranks for inputs are
96
estimated based on the relative magnitude of main and total effects in 30 days of model
simulation. Mean ranks based on the two measures are compared for the three exposure
scenarios.
Figure 5-23 shows the mean rank and corresponding range of ranks based on main and
total effects for each input. The input abbreviations are given in Table A.1 in Appendix A. There
were two key differences between the mean rankings based on the main and total effects of
inputs: (1) mean ranks based on the two sensitivity indices were different for the top sensitive
inputs; and (2) range of ranks associated with each input based on the main effects was wider
compared to that based on the total effects.
In general, the rankings obtained based on the total effects have significant concordance
with the results obtained from other methods. For example, inputs such as PWB, AM, Area, DR,
and FTR are among those with average ranks in the top five for Scenarios I and III, and this is
typically true based on results from other methods. Similarly, inputs that are identified to have
low average ranks (large values of rank), such as Cair, Cb, RN/T, Height, PAI, Dinh, and others, are
consistently identified as unimportant by all of the methods.
However, there are some potentially significant differences in rankings based upon
Sobol’s method depending on whether only main effects are considered or whether interaction
effects are considered. For example, probability of washing body (PB) had a mean rank of 12
with a range of ranks between 2 and 23 in Scenario II based on the relative magnitude of the
main effects. However, PB had a mean rank of 3 with a range of ranks between 1 and 9 based on
the relative magnitude of the total effects. Because in Scenario II the contribution of inputs to the
97
(a) Ranks Based on Main Effects
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
anks
Scenario I Scenario II Scenario III
(b) Ranks Based on Total Effects
0
5
10
15
20
25
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
anks
Scenario I Scenario II Scenario III
Figure 5-23. Comparison of Mean Ranks and Range of Ranks Based on the Sobol’s Method for Inputs in 30 Days for the Three Temporal Scenarios Based on: (a) Main Effects; and (b) Total Effects.
98
output variance via their main effects was relatively low, and hence, inputs mostly contributed to
the output variance via their nonlinear or interaction effects, a ranking based on the relative
magnitude of the main effects provides misleading results. The differences in rankings of inputs
in the other two exposure scenarios were not that sensitive to the choice of the sensitivity index
as a measure of ranking because the contribution of the main effects to the output variance were
substantially larger in those two scenarios.
Based on the mean total effects, on average, the most sensitive input in Scenario I was
probability of washing body (PWB) with a mean rank of 1.7. The range of ranks for this input was
between 1 and 2. Inputs with secondary sensitivity with respect to daily exposures were residue
decay rate (DR), fraction of chemicals for transfer (FTR), amount of chemical applied (AM), and
body weight (WB). Average ranks for these inputs within 30 days of model simulation were
between 2.6 and 4.6.
In Scenario II, FTR was identified as the most sensitive input with a mean rank of 1.3.
Transfer coefficient via body (TCBody), probability of washing body (PWB), residue decay rate
(DR), exposure duration at target area (EDTarget), and body weight (WB) were identified as inputs
with secondary sensitivity with mean ranks between 3.0 and 4.6.
In Scenario III, PWB was identified as the most sensitive input throughout the entire
month. Inputs with secondary sensitivity included DR, AM, Area, and WB with mean ranks
between 3.0 and 6.0.
5.5.5 Summary of Results Based on Sobol’s Method
Overall, results based on Sobol’s method are qualitatively consistent with those of other
methods. However, Sobol’s method offers an advantage of providing insight into the difference
between the main, linear effect of an input and the total effect that includes all of the possible
99
interactions of a given input with all other inputs. A comparison of the total and main effects
provides some insight into the behavior of the model and the significance of each input when
interactions are accounted for. The Sobol’s algorithm includes a method for reporting results as
a contribution to variance which is a convenient way to summarize and communicate results.
Sobol’s method tends to be much more computationally intensive than methods evaluated in
previous sections.
5.6 Fourier Amplitude Sensitivity Test
This section presents the results based upon the use of FAST to characterize the
sensitivity of the selected model outputs for temporal Scenarios I, II, and III to each of the inputs
of the model. Selected inputs the same as those presented in previous sections are considered as
the focus of discussion here and specific details are given.
Comments are made regarding the contribution of inputs to the output variance (Section
5.6.1). Main and total effect sensitivity indices for selected inputs are given and compared
(Section 5.6.2). Changes in ranking of selected inputs with respect to time are presented (Section
5.6.3). Mean ranks and associated range of ranks in 30 days of model simulation for all inputs
are also provided and compared (Section 5.6.4). Summary of the key findings based on FAST
are discussed (Section 5.6.5).
5.6.1 General Insight Regarding Contribution of Inputs to the Output Variance
For each of the three temporal scenarios, FAST was applied to the model at each time
step. Main and Total effects are estimated for all inputs. The main effect of each input represents
the fractional contribution of the input to the output variance and the sum should equal to one for
linear additive models. However, in the case of non-linearity and interactions between inputs
sum of main effects will be less than one. Figure 5-24 shows the sum of main effects for each of
the three temporal scenarios.
100
(a) Scenario I
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sum
of M
ain
Effe
cts
(b) Scenario II: Incremental Change In Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sum
of M
ain
Effe
cts
(c) Scenario II: Cumulative Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sum
of M
ain
Effe
cts
Figure 5-24. Variation in Sum of Main Effects of Inputs Based on FAST with Respect to Time for: (a) Daily; (b) Incremental; and (c) Cumulative Total Exposure Scenarios.
Figure 5-24 (a) shows the sum of main effects of inputs at each time step in Scenario I.
On average, approximately 60 percent of the output variance was apportioned to the main effects
of inputs within the first 10 days of the model simulation. However, in the last 10 days of the
101
model simulation, the sum of main effects decreased to values between 35 and 45 percent. Thus,
approximately 40 percent of the output variance was because of the nonlinearity and possible
interactions between inputs within the first 10 days of the model simulation, while this amount
increased to values between 55 and 65 percent within the last 10 days of the model simulation.
This result is approximately similar to that obtained from Sobol’s method as shown in Figure 5-
18(a).
Figure 5-24 (b) shows sum of main effects of inputs at each time step in Scenario II. In
Scenario II, contribution of main effects of inputs to the output variance was substantially lower.
For example, approximately between 10 and 15 percent of the output variance was apportioned
to the main effects of inputs in the last 10 days of the model simulation. This is also consistent
with the results from Sobol’s method.
Figure 5-24 (c) shows sum of main effects of inputs at each time step in Scenario III. . In
Scenario III, the contribution of main effects of inputs to the output variance did not typically
change at different time steps of the model simulation and was approximately 60 percent. These
findings were consistent with the results shown in Figure 5-8 for the variation of R2 values with
respect to time and those based on the Sobol’s method in Figure 5-18(c).
5.6.2 Sensitivity Indices
For each of the three temporal scenarios, main and total effect sensitivity indices based
on FAST were estimated at each day for all inputs. Thus, 30 sets of main and total effects were
estimated at each time step of the model simulation. Typically, sensitivity indices varied in time
in the three temporal scenarios. Two examples are provided.
102
Figure 5-25 shows variation in main and total effects with respect to time for residue
decay rate (DR). These results are qualitatively similar to those from all other methods, including
Sobol’s method.
For all three scenarios, DR typically has smaller main and total effects in the earlier time
steps compared to the later time steps of the model simulation. Moreover, the difference between
main effects and total effects representing contribution of DR to the output variance in form of
interactions with other inputs increased in later time steps.
For Scenario I, the difference between total and main effects of DR varied between 0 and
0.16 in the first 10 days of the model simulation in Scenario I, while this value ranged between
0.35 and 0.45 in the last 10 days of the model simulation. Thus, approximately 16% of the
output variance can be apportioned to interaction between DR and other inputs at Day 10 and 40
percent at Day 30. The main effect of 0.2 at Day 30 is comparable to, although slightly lower,
than the main effect estimated by Sobol’s method.
In Scenario II, DR typically contributed to the output variance via interactions with other
inputs since the corresponding main effects were negligible in each time step. For example, on
Day 30, although 59% of the output variance was apportioned to overall effect of DR, only 3% of
this amount was because of the direct contribution of DR to the output variance via its main
effect. Thus, 97% of the overall contribution of DR to the output variance was because of
interaction of DR with other inputs in the model. However, individual interaction effects are not
quantifiable based on the sensitivity index, and thus it is not possible to have additional detail on
which combinations of inputs are contributing to the large interaction effect.
Figure 5-26 shows variation in main and total effects with respect to time for fraction of
chemicals available for transfer (FTR). These results are qualitatively similar to those of other
103
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
Figure 5-25. Main and Total Effects for Residue Decay Rate Based on FAST for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-
Cumulative Exposure.
104
(a) Scenario I: Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(b) Scenario II: Incremental Change in Daily Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
(c) Scenario III: Cumulative Exposure
0
0.2
0.4
0.6
0 5 10 15 20 25 30Time (day)
Sen
sitiv
ity In
dex
Main EffectTotal Effect
Figure 5-26. Main and Total Effects for Fraction of Chemicals Available for Transfer Based on
FAST for: (a) Scenario I-Daily Exposure; (b) Scenario II-Incremental Change in Daily Exposure; and (c) Scenario III-Cumulative Exposure.
105
methods, including Sobol’s method. Typically, the main and total effects associated with FTR
decreased monotonically with respect to time in Scenarios I and III. In Scenario II, although
main effect of FTR decreased in time, the contribution of interactions involving FTR to the output
variance typically increased, and hence, the total effect of the input remained constant around an
approximate value of 0.35.
5.6.3 Ranking for Selected Inputs
At each time step, inputs were ranked based on the relative magnitude of the total effects.
The rank of one is assigned to an input with highest total effect. As discussed for the Sobol’s
method, the total effects of inputs provides a complete characterization of the sensitivity but does
not enable attribution of the specific interactions that are contributing the most to the overall
interaction effect. However, the total effects are more reliable than the main effects in order to
investigate the overall effect of each single input on the output. In Section 5.6.4, mean ranks
based on average main effects and average total effects are compared to evaluate possible
misleading insight with respect to sensitivity based on main effects. Typically, similar pattern
with respect to change in ranks of inputs compared to previous methods was observed.
Figure 5-27 (a) shows variation in ranks of DR with respect to time for the three temporal
scenarios. DR typically showed lower sensitivity earlier in the month, while its sensitivity
increased later in the month for the three temporal scenarios. For example, while DR had a range
of ranks between 16 and 2 in the first 7 days of the model simulation, it was typically identified
among the top two sensitive inputs through out the rest of the month in Scenario II.
Figure 5-27 (b) shows an example in which selected input had relatively higher
sensitivity in the earlier time steps of the model simulation. FTR had higher sensitivity within the
first 5 days of the model simulation compared to the rest of the month for daily and cumulative
106
(a) Residue Decay Rate
0
5
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
kScenario I Scenario II Scenario III
(b) Fraction of Chemicals Available for Transfer
0
5
10
15
20
25
1 5 10 15 20 25 30
Time (day)
Ran
k
Scenario I Scenario II Scenario III
Figure 5-27. Variation of Ranks with Respect to Time for the Three Temporal Scenarios for: (a) Residue Decay Rate; and (b) Fraction of Chemicals Available for Transfer Based on FAST.
exposure scenarios. However, FTR was identified among the top two sensitive inputs through the
whole month in Scenario II.
In general, the results from FAST for the ranking over time of these two selected inputs
were similar to those obtained from other methods. The key insights, such as that DR is
unimportant early in the month but most important late in the month, and that FTR is most
important early in the month and of moderate importance late in the month, are the same based
on all methods.
5.6.4 Comparison of Mean Ranks
This section presents the mean rank associated with each input within the time period of
the model simulation. The mean rank represents the arithmetic average of ranks for each input.
Lower Rank Implies Greater Sensitivity
107
The range of ranks also is given for each input. The range of ranks represents minimum and
maximum ranks for the input within 30 days of model simulation. Mean ranks for inputs are
estimated based on the relative magnitude of main and total effects in 30 days of model
simulation. Mean ranks based on the two measures are compared for the three exposure
scenarios.
Figure 5-28 shows the mean rank and corresponding range of ranks based on main and
total effects for each input. The input abbreviations are given in Table A.1 in Appendix A. There
were two key differences between the mean rankings based on the main and total effects of
inputs: (1) mean ranks based on the two sensitivity indices were different for the top sensitive
inputs; and (2) range of ranks associated with each input based on the main effects was wider
compared to that based on the total effects.
In general, the rankings based on the total effect are similar to those obtained from other
methods. The rankings based on the total effect are more reliable than those based only on the
main effect, because of the effect of nonlinearities and interactions. For example, in Scenario II
probability of washing body (PWB) had a mean rank of 11.5 with a range of ranks between 2 and
22 based on the relative magnitude of the main effects. However, based on the total effects PWB
had a higher mean rank of 4.2 with a narrower range of ranks between 2 and 8. Because in
Scenario II the contribution of inputs to the output variance via their main effects was relatively
low, and hence, inputs mostly contributed to the output variance via their nonlinear or interaction
effects, ranking based on the relative magnitude of the main effects provided misleading results.
The differences in rankings of inputs in the other two exposure scenarios were not as sensitive to
the choice of the sensitivity index as a measure of ranking because the contribution of the main
effects to the output variance were substantially larger in those two scenarios.
108
(a) Rank Based on Main Effects
0.0
5.0
10.0
15.0
20.0
25.0
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
ank
Scenario I Scenario II Scenario III
(b) Ranks Based on Total Effects
0.0
5.0
10.0
15.0
20.0
25.0
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
ank
Scenario I Scenario II Scenario III
Figure 5-28. Comparison of Mean Ranks and Range of Ranks Based on the FAST for Inputs in 30 Days for the Three Temporal Scenarios Based on: (a) Main Effects; and (b) Total Effects.
109
Based on the mean total effects, on average, the most sensitive input in Scenario I was
probability of washing body (PWB) with a mean rank of 2.1. The range of ranks for this input was
between 1 and 3. Inputs with secondary sensitivity with respect to daily exposures were fraction
of chemicals for transfer (FTR), residue decay rate (DR), amount of chemical applied (AM), and
body weight (WB). Average ranks for these inputs within 30 days of model simulation were
between 2.8 and 5.8.
In Scenario II, FTR was identified as the most sensitive input with a mean rank of 1.4.
Residue decay rate (DR), transfer coefficient via body (TCBody), probability of washing body
(PWB), exposure duration at target area (EDTarget), and body weight (WB) were identified as inputs
with secondary sensitivity with mean ranks between 3.5 and 4.8.
In Scenario III, PWB was identified as the most sensitive input with a mean rank of 1.4.
Inputs with secondary sensitivity included DR, AM, Area, and WB with mean ranks between 2.8
and 5.8.
5.6.5 Summary of Results Based on FAST
Summary of key insights and findings based on FAST include:
• The magnitude of main and total sensitivity indices typically changed for inputs with
respect to time.
• Main and Total Effects of inputs with a monthly sampling strategy typically: (1)
declined monotonically; or (2) increased monotonically with respect to time.
• Main effect for inputs with a daily sampling strategy typically declined monotonically
with respect to time.
• Some inputs with a daily sampling strategy had increase in their contribution to the
output variance via later in the month in the form of interactions with other inputs.
110
• Total effects provided a better sensitivity index for ranking inputs specially when
there is substantial non-linearity in the model.
• Results for Scenario II for the top sensitive inputs were typically different from those
in the other two temporal scenarios.
5.7 Comparison of Results for Selected Sensitivity Analysis Methods
This section presents a comparison between the results from selected sensitivity analysis
methods. Several of the selected sampling-based methods are based on either a linearity
assumption (e.g., sample correlation, sample regression) or monotonic relationship between
output and inputs (e.g., rank correlation, rank regression). For the case of ANOVA, which does
not impose any specific model structure assumptions, only the main effects of inputs were
considered in the analysis. Hence, possible interaction effects between inputs were put aside.
This is not an inherent limitation of ANOVA, but is simple a limitation of the case studies. The
use of ANOVA in order to quantify the main effects only was intended to represent a relative
simple application of ANOVA. Both Sobol’s method and FAST account for main effects and
interaction effects. Although there is a limitation in the algorithm used here for FAST, as
described in Chapter 3, the performance of FAST appears to be reasonable in comparison to
other methods, and thus the performance of FAST does not appear to have been significantly
comprised by the presence of frequencies that were not commensurate. Of course, whether
FAST would continue to perform robustly in other model applications with this limitation would
require additional study. This is addressed in the recommendations of Chapter 6.
One key question is with respect to reliability of rankings based on the selected sampling-
based techniques when compared to those based on the variance-based methods such as FAST
and the Sobol’s method. The selected variance-based techniques are global sensitivity analysis
111
methods which do not assume any functional form for the relationship between the output and
inputs and can be applied to models that are non-linear and/or non-monotonic. Among the three
alternative exposure scenarios defined in this chapter, Scenario II provided the most non-linear
response in the model. Typically, R2 values from application of standardized linear regression
based on the sampled and rank transformed data to Scenario II were substantially low in different
time steps of the model simulation. Thus, Scenario II provides an appropriate testbed for
evaluation of selected sensitivity analysis methods.
Figure 5-29 shows mean ranks of inputs in 30 days based on different sensitivity analysis
methods for Scenario II. Range of ranks for each input is also provided. Each range of rank
represents minimum and maximum rank of the input in 30 days of model simulation. Mean ranks
are shown for two categories of inputs: (a) inputs with a monthly sampling frequency; and (b)
inputs with a daily sampling frequency. There are two key differences between the results based
on the sampling-based and variance-based methods: (1) typically, variance-based method
provided lower mean ranks for the most sensitive inputs, particularly those with a monthly
sampling frequency; and (2) the range of ranks for each input based on the variance-based
methods was narrower compared to that based on the sampling-based methods. One example is
provided for each of the inputs with monthly and daily sampling frequencies.
Probability of washing body (PB) is an input with a monthly sampling frequency. The
mean ranks for PB based on the sampling-based methods were between 10.4 and 14.5 with a
112
(a) Inputs with a Monthly Sampling Strategy
0.00
5.00
10.00
15.00
20.00
25.00
Cai
r
Cb
WB k
PH
PW
B
AM
Are
a
RN
/T
Hei
ght
DR
Inputs
Mea
n R
ank
PCA SCA SRA RRA ANOVA Sobol FAST
(b) Inputs with a Daily Sampling Frequency
0.00
5.00
10.00
15.00
20.00
25.00
PA
I
Din
h
HW
BW LH LB E
f
TCha
nd
TCbo
dy
ED
targ
et
ED
non-
targ
et
FTR
FBU HF
Inputs
Mea
n R
ank
PCA SCA SRA RRA ANOVA Sobol FAST
Figure 5-29. Comparison of the Mean Ranks and Corresponding Range of Ranks in 30 Days of Model Simulation Based on Pearson Correlation Analysis (PCA), Spearman Correlation Analysis (SCA), Sample Regression Analysis (SRA), Rank Regression Analysis (RRA), Analysis of Variance (ANOVA), Sobol’s Method, and FAST for: (a) Inputs with Monthly Sampling Frequencies; and (b)
Inputs with Daily Sampling Frequencies in Scenario II.
113
range of ranks between 5 and 25. However, the mean ranks for PB based on FAST and Sobol’s
methods were 3 and 4.2, respectively, with a range of ranks between 1 and 9.
Maximum dermal load via body (LB) is an input with a daily sampling frequency. The
mean ranks for LB based on the sampling-based methods were between 10.0 and 14.9 with
arange of ranks between 2 and 25. However, the mean ranks for PB based on FAST and Sobol’s
methods were 4.6 and 3.5, respectively, with a range of ranks between 2 and 12.
Thus, in the selected examples, there are some differences in results when comparing
methods. However, when looking at the overall set of results, there are more similarities than
differences. For example, the ranks obtained for inputs such as Cair, Cb, k, Ph, AM, Area, RN/T,
Height, PAI, Dinh, LH, Ef, TCBody, EDtarget, EDnon-target, FTR, and HF were approximately the same
among all methods.
When there were differences between methods, the differences were primarily between
Sobol’s method versus correlation, regression, and ANOVA, or between FAST and correlation,
regression, and ANOVA. In these cases, Sobol’s method and FAST produced similar results
even though the results of these two methods differed from the other five methods. Examples
include WB, PWB, DR, and LB. These examples tend to be for inputs that are highly sensitive
according to Sobol’s and FAST methods and less sensitive according to the other methods. This
comparison is most likely because Sobol’s method and FAST do a better job of more completely
characterizing the full relationship between the output and a given input, including complex
interaction effects. The interaction affects are only incompletely characterize by the rank
correlation and regression effects. Although interactions were not considered with ANOVA,
they could be in future work.
114
An overall implication of the comparison for Scenario II, which is perhaps the most
challenging of the three temporal scenarios with regard to sensitivity, is that the two variance-
based methods appear to produce better and more reliable results than the other methods. For
example, using the more conventional methods, one might fail to properly recognize the
importance of some of the inputs, with PWB serving as perhaps the best example.
Based upon the case studies reported here, key findings regarding each of the sensitivity analysis
methods and recommendations for implementation and additional research are provided in the
next chapter.
115
6. CONCLUSIONS AND RECOMMENDATIONS
In this chapter, conclusions drawn from the associated research work and the results are
presented. Key questions that are raised in Chapter 1 are addressed. Suggestions for future
research directions are provided.
6.1 Main Characteristics of the SHEDS Models Relevant to the Process of Choosing Appropriate Sensitivity Analysis Methods
The main characteristics of the SHEDS-Pesticides model included: (1) non-linearity and
interaction between inputs; (2) saturation points; (3) different input types (e.g., continuous versus
categorical); and (4) aggregation and carry-over effects. A simplified version of the SHEDS-
Pesticides model was developed for evaluation of sensitivity analysis methods. The simplified
model retains three of these main characteristics and does not include categorical inputs. An
ideal sensitivity analysis method should be model independent. Specifically, a sensitivity
analysis method should not require any assumptions regarding the functional form of the risk
model and should be applicable to different model formulations. Some methods are considered to
be global and model-independent. However, some methods, including many that are commonly
used, are based upon assumptions regarding the functional form of the model. If a sensitivity
analysis method based upon an assumed functional form of a model is applied to a model with
different characteristics, then the results of the sensitivity analysis may not be valid.
6.2 Available Sensitivity Analysis Methods
Sensitivity analysis methods are typically classified into categories based on their scope,
applicability, and characteristics. We classified sensitivity analysis methods into two categories
of local and global techniques based on Saltelli et al. (2000). Examples of commonly used
sensitivity analysis methods are regression-based techniques (e.g., Helton, 1993), variance
decomposition methods (e.g., Saltelli et al., 1999), and scatter plots (e.g., Kleijnen and Helton,
116
1999; Saltelli et al., 2000; Frey and Patil, 2002). We evaluated a series of seven global sensitivity
analysis methods including Pearson and Spearman correlation coefficients, standardized sample
and rank linear regression, analysis of variance, FAST, and the Sobol’s method via cased studies
with the simplified SHEDS-Pesticides model. Each of these methods has key assumptions that
may or may not conform to the key characteristics of the simplified model. Thus, the case
scenarios in this report provided a testbed for evaluation of selected sensitivity analysis methods
with respect to situations in which the underlying assumptions of each technique may be
challenged by the model.
6.3 Sensitivity Analysis Methods for Application to the SHEDS Models
Table 6-1 summarizes the comparison of evaluated sensitivity analysis methods. Based
on the key criteria, methods should: be able to consider simultaneous variation in inputs; be
computationally efficient; have quantitative measures for ranking key inputs; be reproducible; be
able to apportion the output variance to different model inputs; be model independent; and be
robust in practice. A method is considered as computationally efficient if: (1) statistical software
packages are available for implementation of the method; and (2) the number of model inputs
does not affect the required computational resource. Reproducibility means to what extent
repetition of the calculation procedure leads to the same results. Model independence means the
extent in which the level of additivity of non-linearity of the model influences the correctness of
the results. Robustness means reliability of results when key assumptions of the method are not
met.
117
Table 6-1. Comparison of Various Sensitivity Analysis Methods Sensitivity Analysis Method (a)
Criteria for Comparison (b) PCA SCA SRA RRA ANOVA FAST SobolSimultaneous Variation in Inputs ++ ++ ++ ++ ++ ++ ++ Computational Efficiency ++ ++ ++ ++ +/- +/- +/- Quantitative Measure for Ranking ++ ++ ++ ++ ++ ++ ++ Reproducibility +/- +/- +/- +/- +/- ++ ++ Ability to Apportion the Output Variance
No No +/- +/- +/- ++ ++
Model Independence No No No No Yes Yes Yes Robustness +/- +/- +/- +/- +/- ++ ++ Discrete Inputs - - +/- +/- ++ - Number of Inputs ++ ++ ++ ++ +/- - +/-
a PCA: Pearson Correlation Analysis; SCA: Spearman Correlation Analysis; SRA: Standardized Sample Linear Regression Analysis; RRA: Rank Regression Analysis; ANOVA: Analysis of Variance; FAST: Fourier Amplitude Sensitivity Test
b ++: Completely satisfies the criterion; +/-: Does not completely satisfy the criterion; -: Poorly addresses the criterion
All methods evaluated in this work were able to consider simultaneous variation in
inputs. Sampling-based methods, including correlation and regression based methods, ANOVA
and Sobol’s method are based on Monte Carlo simulation of the model, and hence, incorporate
simultaneous variation of inputs as a part of their sampling strategy. FAST is not based on Monte
Carlo. However, values are generated from the domain of variation of inputs using
transformation functions.
Typically sampling-based methods are computationally efficient. Most of these methods
are built-in features of commonly used software tools such as @RiskTM and Crystal BallTM and
also are available in statistical software packages such as SAS® and free software such as R. The
computing resources required for each of these methods are usually proportional to the resources
necessary to perform a typical Monte Carlo probabilistic simulation with the model. With
available computer hardware, the number of model inputs does not substantially affect the
computational resource required for the sampling-based methods. However, for ANOVA the
118
need for defining levels for continuous inputs and classifying sampled values from probability
distributions of inputs into the levels can impose some computational burden for models with
many inputs. There are few software packages for implementation of FAST and the Sobol’s
methods. Although these are powerful methods that offer advantages over commonly used
techniques, their widespread practical application is limited until software becomes available by
which these methods can be easily incorporated into a risk model (Saltelli et al., 2004).
For the case scenarios in this research, algorithms were developed for FAST and Sobol’s
method and coded in Matlab. The algorithm for FAST requires specification of frequencies that
dictate how each input is sampled. As an ideal and desirable specification of frequencies
considered in FAST is that they should be incommensurate. The incommensurate frequencies
typically have an important property that no single frequency in a frequency set can be expressed
as a linear combination of other frequencies (Lu and Mohanty, 2001). If this criterion is not
satisfied by the set of frequencies considered for the inputs, an interference problem will be
introduced. Interference is a problem when information provided by the frequencies is mixed;
hence, the problem could lead to overestimating the sensitivity indices, and hence, the
contribution of inputs to the output variance (Saltelli et al, 2000). In practice, a published table is
used to produce incommensurate frequencies for up to 50 inputs (Cukier et al, 1978; Saltelli et al,
2000). However, there is not a published record of an algorithm for generating a larger number
of incommensurate frequencies for more than 50 inputs.
Because the simplified SHEDS-Pesticides model has 30 time periods for each of the daily
inputs, and because the goal was to generate independent samples in each of the time periods,
there is a need for different frequencies for a given input variable for each time period (day) of
the simulation. Thus, although there are only 14 daily inputs, there is a need to generate 30 × 14
119
(i.e., 420) independent sets of samples. In addition, there are 11 monthly inputs. Thus, the total
number of independently sampled distributions is 431. Thus, there was a need to generate
additional frequencies beyond the set of 50 incommensurate frequencies available in literature.
In order to generate additional frequencies, an algorithm was needed. Since no algorithm was
located in the literature, a simplified algorithm was developed that produces frequencies that are
not repetitive. However, this does not guarantee that the frequencies are incommensurate.
Although the case study results from FAST were comparable to those from Sobol’s method,
even though frequencies that were incommensurate were included as a basis for the simulation,
this does not guarantee that FAST will be robust when applied more generally. For example, it
could simply be that in cases where two or more frequencies were not incommensurate, they
happened to be assigned to inputs that did not have any significant influence on the results.
The implications of the need for incommensurate frequencies for FAST, and the lack of
availability of either data for such frequencies or an algorithm for generating such frequencies
when more than 50 inputs are needed is that more research is needed in order to further develop
or evaluate FAST when many inputs are to be analyzed. For the typical SHEDS model, there
can be many inputs and many different time periods, with possibly different times scales for
various inputs. Thus, a sensitivity analysis method must be able to accommodate a large number
of inputs in order to be applicable to the SHEDS models. This research has represented the first
known application of FAST to a model of this type. Thus, as part of this research, a limitation of
FAST has been identified that does not typically apply to the more routine applications of FAST
reported in the literature. There are two general ways to deal with the limitation on the number
of incommensurate frequencies that are available for FAST. One is to conduct additional
research to develop an algorithm for generating additional such frequencies in whatever numbers
120
are needed for application to a given SHEDS model. A second is to evaluate the robustness of
FAST even if the requirement for incommensurate frequencies is not satisfied, in order to
identify conditions under which the results might be correct as long as interference problems are
minimized.
For Sobol’s method, and in some preliminary analysis with a simple linear model,
Sobol’s method provided accurate sensitivity coefficients. However, in one case in which the
scale of the inputs was changed, the convergence toward accurate values of sensitivity indices
required a substantially larger number of iterations. This comparison suggests that the number of
iterations required for Sobol’s method may be sensitive to the scale of the model inputs or to
whether the variation in the model output is large. In practice, and as shown in Chapter 5,
Sobol’s method provided reasonable results for the simplified SHEDS-Pesticides model in
comparison to other sensitivity analysis methods, and thus a convergence problem was not
suspected there. The output for the simplified SHEDS-Pesticides model has a large range of
variation relative to its mean. A hypothesis is that in this type of situation, Sobol’s method is able
to converge on accurate sensitivity indices with a moderate number of iterations. However, in
order to address this problem, the output of Sobol’s method was normalized, and the normalized
estimates of contribution to variance were found to converge to the correct value more quickly
than an estimate based on the raw sensitivity indices. Thus, the use of normalized sensitivity
measures appears to facilitate more rapid convergence, thereby reducing the needed sample size.
However, the properties of the normalization with respect to the necessary sample size require
additional assessment in order to prepare guidance on how to choose a sample size for Sobol’s
method.
121
There may be variation in the rate at which Sobol's method converges on accurate
values of the sensitivity coefficients, possibly depending on the scale of the inputs or depending
on whether the range of variability in the output is small relative to its mean. However, in future
work the hypothesis should be tested as to whether Sobol's method may require more iterations
to converge on sensitivity coefficients depending on the scale of model inputs, on the variance of
the model output relative to its mean, or the normalization method used to estimate the
contribution to variance.
Sobol’s method is not computationally efficient for models with many inputs. Full sets of
first plus total effect sensitivity indices can be estimated at the cost of )22( +×× knt
simulations, where t is the number of time steps considered in the analysis (e.g., 30 for the case
scenarios in this research), n is the number of bootstrap simulations for estimating the confidence
intervals for sensitivity indices, and k is the number of model inputs. If sensitivity indices for
other effects such as two-way and three-way interactions between inputs are to be estimated, the
computational cost will increase drastically.
All of the selected methods for sensitivity analysis provide quantitative measures for
ranking inputs. The ability to produce quantitative rankings and the ability to evaluate the
statistical significance of the rankings are useful to identify the relative importance of inputs and
the confidence that should be imputed to the rankings. Some methods produce more useful
measures by which to discriminate the importance among similarly ranked inputs. For example,
correlation and regression-based methods provide confidence intervals for sensitivity indices
based on which statistically significant difference between indices associated with different
inputs can be inferred. Sobol’s method includes bootstrap simulation, and hence, is able to
122
identify the standard error associated with the sensitivity indices. However, for FAST no
criterion was identified to evaluate whether sensitivity indices were statistically different.
Results based on the sampling-based techniques are sensitive to the simulation sample
size. With small sample sizes, the results may change based on different set of samples. Frey et
al. (2003) show some case studies in which sensitivity analysis results based on ANOVA
changed substantially when alternative set of samples were used for the analysis. Variance-based
techniques evaluated in this research were typically reproducible. FAST produces the same
mapping from the input domain to the model output space when the number of inputs does not
change. The Sobol’s method incorporates the application of the bootstrap technique in estimation
of sensitivity indices, and hence, can quantify possible uncertainty in those estimates due to the
sample size.
The ability to apportion the output variance to individual inputs and their combinatory
effects has been considered as the key objective of this research. Among the selected sensitivity
analysis methods, Sobol’s method and FAST directly provide insight with respect to this
objective. Sensitivity indices based on these two methods represent the contribution of the
selected individual input or combination of inputs to the output variance. For example, a main
effect of 0.5 for the input x1 indicates that 50% of the output variance can be apportioned to x1.
Correlation-based methods are not able to provide insight with regard to the contribution to
variance. However, regression-based methods can provide some insight in this regard. The
coefficient of determination, R2, represents the amount of output variability that can be captured
by a regression or ANOVA approach. Furthermore, some measure of variation in the output,
such as sum of squares of the output values, can be apportioned to individual terms (e.g., input
terms, interaction terms, or any nonlinear term). However, this issue was not investigated here.
123
The evaluated sampling-based methods are not typically model-independent. These
methods are based upon assumptions regarding the functional form of the model. For example,
sample correlation analysis or linear regression is based upon the assumption of a linear model.
Rank correlation coefficients or rank regression are based upon the assumption of a monotonic
model. ANOVA and the two variance-based methods are model-independent. If a sensitivity
analysis method based upon an assumed functional form of a model is applied to a model with
different characteristics, then the results of the sensitivity analysis may not be valid. For
example, if method based upon linearity, such as sample correlation analysis, is applied to a
nonlinear or non-monotonic model, then the insights regarding sensitivity could be inaccurate or
invalid.
Results based on the sampling-based methods of correlation and regressions are typically
perceived, on a theoretical basis, not to be robust because these methods are not model
independent. However, in practice, the results obtained from these methods, and particularly for
correlation coefficients, were comparable to those from ANOVA, which is considered to be a
model-independent technique. Thus, in practice, with at least some kinds of models, correlation
and regression models can enable useful insights regarding sensitivity. For ANOVA, although
this method does not assume any functional form, results can be sensitive to the definition of
levels and the number of data within each level after classification of sampled data (Neter et al.,
1996; Kleijnen and Helton, 1999).
The emphasis of this project has been on evaluation of methods that can provide insight
with respect to contribution of inputs to the output variance. FAST and the Sobol’s methods are
among the variance-based techniques that can apportion the output variance to individual effects
of inputs along with their combinatory contributions. However, and due to computationally
124
intensiveness of these two methods, simpler methods can be used as screening techniques in the
earlier stage of the analysis to identify those inputs that are typically unimportant. Our case study
results typically showed that all methods had agreement in identification of unimportant inputs.
Such inputs with a small percentage of contribution to the output variance can be frozen to any
value within their range of variation (e.g., mean). This process contributes substantially to the
model simplification and applicability of resource intensive methods such as FAST and Sobol’s
method to models with several inputs.
In general, five of the methods, including sample correlation, rank correlation, sample
regression, rank regression, and ANOVA provided results that were comparable to each other.
With regard to correlation and regression, the two rank-based methods are more robust and
reliable than the two sample-based methods. The differences in results between rank and
sample-based methods are somewhat analogous to the comparison of total versus main effect in
methods such as Sobol’s and FAST; however, they are not as general in capturing interaction
effects as are the latter two methods. A comparison of rank versus sample based results typically
provides some insight regarding the importance of nonlinearity in evaluating sensitivity results,
as was illustrated in several case studies.
Both Sobol’s and FAST methods produced similar results to each other, but tended to
differ from the other methods. In particular, for Scenario II, which was perhaps the most
challenging testbed for sensitivity analysis methods because of the large importance of
interaction effects, Sobol’s and FAST methods identified some inputs as more important than did
the other five methods. This difference is most likely because Sobol’s and FAST methods are
more capable of capturing the effect of interactions than are the other methods. However, the
125
application of regression and ANOVA in the case studies here did not take into account
interaction effects, which could be included.
The regression and ANOVA methodologyies have ways of dealing with user-specified
interaction effects. In the case of regression, the interaction effect has to be evaluated with
respect to a particular functional form of a regression model, whereas for ANOVA it is not
necessary to specify a specific functional form. However, even if interaction effects are
accounted for in regression and ANOVA, a practical advantage of Sobol and FAST is that all
interactions are systematically accounted for as an inherent part of the methodology.
Based on the summary results for comparison of the selected sensitivity analysis methods
in Table 6-1, it follows that Sobol’s method produced the best overall performance. FAST is
also a promising method but the current limitations pertaining to development of frequencies that
are used as the basis for sampling require additional research before this method could be
recommended for routine application to the SHEDS models. The other methods, such as
correlation, regression, and ANOVA, are useful as screening methods and are capable of
providing useful results for models that are approximately linear. However, they do not directly
quantify contribution to variance in the manner that Sobol’s method does.
6.4 Recommendations
Key recommendations are as follows:
(1) Sobol’s method is identified as a promising method for apportioning the variance in a
model output to individual inputs and to the interaction effects among inputs.
Therefore, the application of Sobol’s method to a larger, or more complex, model is
recommended as the next step toward more routine applications of this method to
models such as SHEDS. In future work the hypothesis should be tested as to whether
126
Sobol's method may require more iterations to converge on sensitivity coefficients
depending on the scale of model inputs, on the variance of the model output relative
to its mean, or the normalization method used to estimate the contribution to variance.
Furthermore, there is a need for guidance on how to choose a sample size for SHEDS.
There are also some applicability issues that must be addressed, that are covered in
other recommendations below, that are general for many methods including Sobol’s.
(2) The correlation, regression, and ANOVA methods were found to produce comparable
results in most cases, and they enabled insight into complex temporal issues,
nonlinearities, and competition among inputs with respect to importance. These
methods appear to be useful in practice for identifying inputs that do not matter and
thus could be used as a screening step prior to application of a more refined method
such as Sobol’s. Furthermore, a comparison of sample versus rank-based results for
either correlation or regression provides insight into the importance of nonlinearity in
the model and implications for sensitivity. An area of research for regression-based
methods is to explore the use of sum of squares of the output values, in order to
apportioned the response of the output to individual terms (e.g., input terms,
interaction terms, or any nonlinear term) in the context of the SHEDS model.
ANOVA could be extended to include interaction effects in a model independent
manner, implying some advantage over the inherently model-dependent approach of
correlation or regression techniques. In general, it can be useful to apply and
compare results from sample and rank based methods, or to apply and compare
ANOVA results with one of the correlation or regression based approaches. Thus,
127
were possible to apply two or more methods, these methods can be used in
combination or as screening methods.
(3) We did not consider any dependency structure between inputs in the simplified
SHEDS-Pesticides model. However, for cases that inputs have dependency structures
such as correlation, sensitivity analysis methods should be evaluated and, if
necessary, modified in order to incorporate such dependencies in the analysis.
Sensitivity analysis methods should be evaluated with respect to their functionality in
the case of inputs with dependency. Approaches should be explored for quantification
of different dependency structures between inputs and possible modification of
sensitivity analysis methods for such cases.
(4) For the selected variance-based techniques, including Sobol’s method and FAST,
algorithms were provided for estimation of first and total effects of each input.
Although the total effects of inputs are more reliable than the first-order (main
effects) indices, they do not provide a complete characterization of the sensitivity in
that they do not enable a full explanation of the major cause of interactions. Thus,
optimized algorithms should be provided in order to break down the total effect into
interaction effects between inputs, such as pairwise, three-way combinations, and so
on.
(5) The simplified SHEDS-Pesticides model assumes that inputs are the only sources of
variability in the model, and hence, the whole output variance can be apportioned to
inputs. However, there may be stochastic sources of variability in the model that are
not included in the form of an input in the model. For example, data might be
sampled from a database for joint values of several inputs, in order to capture intra-
128
individual dependencies. Selected variance-based sensitivity analysis methods may
fail to provide accurate estimates of sensitivity indices specifically with respect to the
total effect of each input, when stochastic sources of variability are available in the
model. Thus, these methods should be evaluated and modified as needed to be
applicable to models with such sources of variation in the response.
(6) FAST is a promising method for estimating contribution to variance. However,
additional research is recommended in several areas before this method can be
considered for application to models such as SHEDS. One is to conduct additional
research to develop an algorithm for generating additional such frequencies in
whatever numbers are needed for application to a given SHEDS model. A second is
to evaluate the robustness of FAST even if the requirement for incommensurate
frequencies is not satisfied, if a suitable algorithm for generating the needed number
of incommensurate frequencies is not achieved, in order to identify conditions under
which the results might be correct as long as interference problems are minimized. A
third is to evaluate the applicability of FAST with respect to model inputs that are
categorical or discrete.
(7) Future work could explore normalization algorithms for summarizing the fractional
contribution of each input or interaction effect to the sum of sensitivity indices, and
compare the results with those obtained from Sobol’s method or FAST, in order to
determine whether there are practical alternatives to Sobol’s method that might be
more readily available or easier to apply.
(8) This study has focused on the objective of quantifying the contribution to variance of
a model output. If the assessment objective changes, then the choice of an
129
appropriate sensitivity analysis method can also change. For example, if the objective
is to identify key factors influencing high exposures, then a method such as
Categorical and Regression Trees (CART) should be considered. In general, the
results and findings here should not be interpreted to preclude the use of other
methods that might be more appropriate to a particular assessment objective.
131
REFERENCES
Agro, K. E., C. A. Bradley, and N. Mittmann (1997), "Sensitivity Analysis in Health Economic
and Pharmacoeconomic Studies - an Appraisal of the Literature," Pharmacoeconomics,
11(1):75-88.
Archer, G., A. Saltelli, and I. M. Sobol (1997), “Sensitivity Measures, ANOVA-Like
Techniques, and the Use of Bootstrap,” Jurnal of Statistical Computing and Simulation,
58(1): 99-120.
Baker, S., D. Ponniah, and B. Klubek (1999), “Survey of Risk Management in Major UK
Companies,” Journal of Food Protection, 62(9): 1050-1053.
Baniotopoulos, C. C. (1991), "A Contribution to the Sensitivity Analysis of the Sea-Bed
Structure Interaction Problem for Underwater Pipelines," Computers & Structures,
40(6):1421-1427.
Beck, M. B., J. R. Ravetz, and L. A. Mulkey (1997), "On the Problem of Model Validation for
Predictive Exposure Assessments," Stochastic Hydrology and Hydraulics, 11(3):229-254.
Cheng, T. C. E. (1991), "EPQ with Process Capability and Quality Assurance Considerations,"
Journal of the Operational Research Society, 42(8):713-720.
Cukier, R. I., C. M. Fortuin K. E. Shuler, et al. (1973), ”Study of the Sensitivity of Coupled
Reaction Systems to Uncertainties in Rate Coefficients: Part I Theory,” Journal of
Chemical Physics, 59, 3873-3878.
Cukier, R. I., J. H. Schaibly, and K. E. Shuler (1975), “Study of the Sensitivity of Coupled
Reaction Systems to Uncertainties in Rate Coefficients: Part III Analysis of
Approximation,” Journal of Chemical Physics, 63(5), 1140-1149.
132
Cukier R. I. , H. B. Levine, and K. E. Shuler (1978), “Nonlinear Sensitivity Analysis of Multi-
Parameter Model Systems,” Journal of Computational Physics, 26(1): 1-42.
Cullen, A. C, and H. C. Frey (1999). Probabilistic Techniques in Exposure Assessment. Plenum
Press, New York, USA.
DeGroot, M. H. (1986). Probability and Statistics, 2nd Edition. Addison-Wesley, MA.
Devore, J. L., and R. Peck (1996). Statistics: The Exploration and Analysis of Data. 3rd Edition.
Brooks/Cole Publishing Company: London, England.
Devroye, L. (1986). Non-Uniform Random Variant Generation. Springer-Verlag, New York.
Doll, J. D., and D. L. Freeman (1986), “Randomly Exact Methods,” Science, 234:1356-1360.
Fishman, G. S. (1996). Monte Carlo: Concepts, Algorithms, and Applications. Springer Verlag,
New York.
Fraedrich, D., and A. Gildberg (2000), “A Methodological Framework for the Validation of
Predictive Simulations,” European Journal of Operational Research, 124(1): 55-62.
Frey, H. C. (1992), “Quantitative Analysis of Uncertainty and Variability in Environmental
Policy Making,” American Association for Advancement of Science, Washington DC.
Frey, H. C. (1997). Variability and uncertainty in highway vehicle emission factors. In Emission
Inventory: Planning for the Future (held October 28-30 in Research Triangle Park, NC),
Air and Waste Management Association, Pittsburgh, Pennsylvania, pp.208-219.
Frey, H. C., R. Baharkivar, and J. Zheng (1999). Quantitative analysis of variability and
uncertainty in emission estimations. Final Report, Prepared by North Carolina State
University for Office of Air Quality Planning and Standards, U. S. Environmental
Protection Agency, Research Triangle Park, NC.
133
Frey, H. C., A. Mokhtari, and J. Zheng (2004). Recommended practice regarding selection,
application, and interpretation of sensitivity analysis methods applied to food safety
process risk models. Prepared by North Carolina State University for Office of Risk
Assessment and Cost-Benefit Analysis, U.S. Department of Agriculture. Washington DC.
www.ce.ncsu.edu/risk.
Frey, H. C., and R. Patil (2002), “Identification and Review of Sensitivity Analysis Methods,”
Risk Analysis, 22(3): 553-577.
Frey, H. C., D. S. Rhodes (1996), “Characterizing, Simulating and Analyzing Variability and
Uncertainty: An Illustration of Methods Using an Air Toxics Emissions Example,”
Human and Ecological Risk Assessment, 2(4), 762-797.
Frey, H. C., J. Zheng (2000). Methods and example case study for analysis of variability and
uncertainty in emission estimation (AUVEE). Prepared by North Carolina State
University for Office of Air Quality Planning and Standards, U.S. Environmental
Protection Agency, Research Triangle Park, NC.
Frey, H. C., J. Zheng (2002), “Probabilistic Analysis of Driving Cycle-Based Highway Vehicle
Emission Factors,” Environmental Science and Technology, 36(23), 5184-5191.
Gardiner, C. W. (1983). Handbook of Stochastic Methods for Physics, Chemistry and the Natural
Sciences. Springer-Verlag, New York.
Gibbons, J. D. (1985). Nonparametric Statistical Inference. Marcel Dekker, Inc, New York and
Basel, 2nd edition.
Hahn, G. J., and S. S. Shapiro (1967). Statistical Models in Engineering. Wiley Classics Library,
John Wiley and Sons: New York.
134
Hamed, M. M., J. P. Conte, and P. B. Bedient (1995), “Probabilistic Screening Tool for Ground-
Water Contaminant Assessment,” Journal of Environmental Engineering, 121(11):767-
775.
Harrison, K. W. (2002). Environmental and Water Resource Decision Making Under
Uncertainty. Ph.D. Dissertation, North Carolina State University.
Helton, J. C. (1993), "Uncertainty and Sensitivity Analysis Techniques for Use in Performance
Assessment for Radioactive-Waste Disposal," Reliability Engineering and System Safety,
42(2-3): 327-367.
Helton, J. C., and R. J. Breeding (1993), "Evaluation of Reactor Accident Safety Goals,"
Reliability Engineering & System Safety, 39(2):129-158.
Iman, R. L., and W. J. Conover (1980), “Small Sample Sensitivity Analysis Techniques for
Computer Models, with an Application to Risk Assessment,” Communications in
Statistics, Part A. Theory and Methods, 17:1749-1842, 1980.
Isukapalli, S. S., A. Roy, P. G., and P. G. Georgopoulos (1998). Stochastic response surface
methods (SRSMs) for uncertainty propagation: Application to environmental and
biological systems. Online report. Rutgers University, Piscataway, NJ.
http://www.ccl.rutgers.edu/~ssi/srsmreport/index.html
Jones, R. N. (2000),”Analyzing the Risk of Climate Change Using an Irrigation Demand
Model,” Climate Research, 14(2): 89-100.
Karamchandani, A., and C. A. Cornel (1992), “Sensitivity Estimation Within First and Second
Order Reliability Methods,” Structural Safety, 11(2):95-107.
Kalos, M. H., and P. A. Whitlock (1986). Monte Carlo Methods : Basics. John Wiley & Sons,
New York.
135
Kendall, M. G., and J. D. Gibbons (1990). Rank Correlation Methods. Fifth Edition. Charles
Griffin: London.
Kewley, R. H., M. J. Embrechts, and C. Breneman (2000), "Data Strip Mining for the Virtual
Design of Pharmaceuticals with Neural Networks," IEEE Transactions on Neural
Networks, 11(3):668-679.
Kini, M. D., H. C. Frey (1997). Probabilistic evaluation of mobile source air pollution: Volume
1, Probabilistic modeling of exhaust emissions from light duty gasoline vehicles.
Prepared by North Carolina State University for Center for Transportation and the
Environment, Raleigh, December.
Kleijnen, J. P. C. (1995), “Verification and Validation of Simulation-Models,” European Journal
of Operational Research, 82(1): 145-162.
Kleijnen, J. P. C., and J. C. Helton (1999), “Statistical Analyses of Scatterplots to Identify
Important Factors in Large-Scale Simulations, 1: Review and Comparison of
Techniques,” Reliability Engineering and System Safety, 65(2): 147-185.
Kleijnen, J. P. C, and R. G. Sargent (2000), “A Methodology for Fitting and Validating
Metamodels in Simulation,” European Journal of Operational Research, 120(1): 14-29.
Krishnaiah, P. R. (1981). Analysis of Variance. Elsevier: New York.
Limat, S., M. C. Woronoff-Lemsi, E. Deconinck, et al. (2000), "Cost-Effectiveness of Cd34(+)
Dose in Peripheral Blood Progenitor Cell Transplantation for Non-Hodgkin's Lymphoma
Patients: a Single Center Study," Bone Marrow Transplantation, 25(9):997-1002.
Loh, W. H (1996), “On Latin Hypercube Sampling,” Annals of Statistics, 24(5): 2058-2080.
Lu, R., Y. Luo, and J. P. Conte (1994), “Reliability Evaluation of Reinforced Concrete Beam,”
Structural Safety, 14:277-298.
136
Manheim, L. M. (1998), "Health Services Research Clinical Trials: Issues in the Evaluation of
Economic Costs and Benefits," Controlled Clinical Trials, 19(2):149-158.
McCamley, F., and R. K. Rudel (1995), "Graphical Sensitivity Analysis for Generalized
Stochastic Dominance," Journal of Agricultural and Resource Economics, 20(2):403-
403.
Mokhtari, A., and H. C. Frey (2005). “Recommended Practice Regarding Selection of Sensitivity
Analysis Methods Applied to Microbial Food Safety Models,” Human and Ecological
Risk Assessment, 11(3): 1-16.
Morgan, M. G., and M. Henrion (1990). Uncertainty: A Guide to Dealing with Uncertainty in
Quantitative Risk and Policy Analysis. New York: Cambridge University Press.
Morgan, M. G., S. C. Morrise, M. Henrion, et al. (1984), “Technical Uncertainty in Quantitative
Policy Analysis- A Sulfur Air Pollution Example,” Risk Analysis, 4(3): 201-213.
Neter, J., M. H. Kutner, C. J. Nachtsheim, et al. (1996). Applied Linear Statistical Models,
Fourth Edition. McGraw-Hill, Chicago, IL, USA.
Oh, B. H., and I. H. Yang (2000), "Sensitivity Analysis of Time-Dependent Behavior in PSC
Box Girder Bridges," Journal of Structural Engineering-ASCE, 126(2):171-179.
Phillips, A., D. Janies, and W. Wheeler (2000), “Multiple Sequence Alignment in Phylogenetic
Analysis,” Molecular Phylogenetics and Evolution, 16(3):317-330.
Pollak, A. K., P. Bhave, J. Heiken, et al. (1999). Investigation of emission factors in the
California EMFAC7G Model, PB99-14718INZ. Prepared by ENVIRON International
Corp, Novato, CA, for coordinating Research Council, Atlanta, GA.
Price, P. S., F. C. Chaisson, M. Koontz, et al. (2003). Construction of a Comprehensive
Chemical exposure framework using person oriented modeling. Developed for the
137
Exposure Technical Implementation Panel, American Chemistry Council, Contract #
1388. Environmental and Occupational Health Science Institute, Rutgers University.
Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method. John Wiley & Sons, New
York.
Saltelli, A., and R. Bolado (1998), “An Alternative Way to Compute Amplitude Sensitivity Test
(FAST),” Computational Statistics and Data Analysis, 26: 445-460.
Saltelli, A., S. Tarantola, and K. P. Chan (1999), “A Quantitative Model-Independent Method for
Global Sensitivity Analysis of Model Output,” Technometrics, 41, 39-56.
Saltelli, A., K. Chan, and M. Scott (2000). Sensitivity Analysis: Probability and Statistics Series.
John Wiley & Sons, New York, USA.
Saltelli, A. (2002), “Making Best Use of Model Evaluation to Compute Sensitivity Indices,”
Computer Physics Communications, 145: 280-297.
Schaibly, J. H., and K. E. Shuler (1973), “Study of the Sensitivity of Coupled Reaction Systems
to Uncertainties in Rate Coefficients. Part II, Applications,” Journal of Chemical Physics,
59(6): 3879-3888.
Siegel, S., and N. J. Castellan (1988). Nonparametric Statistics for the Behavioural Sciences (2nd
edn), McGraw-Hill, New York.
Sobol, I.M. (1993), “Sensitivity Estimates for Nonlinear Mathematical Models,” Mathematical
Modeling and Computation, 1(4):407-414.
Sobol, I. M. (1994). A Primer for the Monte Carlo Method. CRC Press.
Stein, M. (1987), ”Large Sample Properties of Simulations Using Latin Hypercube Sampling,”
Technometrics, 29(2):143-151.
138
U.S. EPA (1996),”Summary Report for the Workshop on Monte Carlo Analysis,” EPA/630/R-
96/010, Risk Assessment Forum, Washington, DC.
U.S. EPA (1997),”Guiding Principles for Monte Carlo Analysis,” EPA/630/R-97/001, Risk
Assessment Forum, Washington, DC.
U.S. EPA (2000), “First-Generation Multimedia, Multipathway Exposure Models,” NERL
Research Abstract, National Exposure Research Laboratory, Research Triangle Park, NC,
September 2000.
Ward, M. P., and T.E. Carpenter (1996), “Simulation Modeling of the Effect of Climatic Factors
on Bluetongue Virus Infection in Australian Cattle Herds - 1. Model Formulation,
Verification and Validation,” Preventive Veterinary Medicine, 27(1-2):1-12.
Wilson, R., and E. A. Crouch (1981),”Regulations and Carcinogens,” Risk Analysis, 1(1):47-57.
Zartarian, V. G., H. Ozkaynak, J. M. Burke, et al. (2000), “A Modeling Framework for
Estimating Children’s Residential Exposure and Dose to Chlorpyrifos via Dermal
Residue Contact and Non-Dietary Ingestion,” Environmental Health Perspective, 108(4):
505-514.
139
APPENDIX A
A.1 Simplified SHEDS-Pesticide Model
This section provides the key equations in the simplified SHEDS-Pesticides model along
with the probability distributions of model inputs. Figure A.1 shows a schematic diagram of the
simplified model. The final exposure is estimated based on three pathways as:
iIngestioniDermaliInhalationiTotal EEEE ,,,, ++= (A.1) Where, ETotal, i = Total exposure at time step i, i = 1, 2,…,30 (days) [µg/kg]
EInhalation, i = Exposure via inhalation pathway at time step i [µg/kg]
EDermal, i = Exposure via dermal pathway at time step i [µg/kg]
EIngestion, i = Exposure via non-dietary ingestion pathway at time step i [µg/kg]
i = 1, 2,…, 30; represents the number of days since application of the
pesticide
Equations corresponding to each of the pathways including inhalation, dermal, and ingestion are
presented and briefly explained.
Inhalation
Indoor inhalation exposure at each time step is estimated as:
B
iinhBViiInhalation W
DRPAICE ,
,
×××= (A.2)
Where,
EInhalation, i = Exposure via inhalation pathway at time step i [µg/kg]
iC = modeled or measured airborne concentration of pollutant in air at
the ith day [µg/m3]
140
Figure A.1. Schematic Diagram of the Simplified SHEDS Model.
bairi CikExpCC +×−×= )( (A.3)
Cair = Concentration of applied pesticides [µg/m3]
Cb = Background concentration of pesticides [µg/m3]
k = Decay Rate [1/day]
PAI = Physical activity index [-]
Dinh,i = Inhalation Exposure duration time at the ith day [min]
WB = Body weight [kg]
RBV = Basal ventilation rate [m3/min] as:
1440)11.2095.0(434.1 +×
×= BBM
WR (A.4)
Average exposure over exposure duration:
m
EE
m
iiInhalation
Inhalation
∑== 1
,
(A.5)
Where,
141
m = Total duration of exposure considered in the scenario (e.g., m = 30
days)
Dermal
Exposure from dermal pathway consists of exposure from hand and body (non-hand).
Key characteristics of the dermal pathway include: (1) carryover of exposure from prior event or
prior day; (2) adjustment for maximum dermal loading; (3) adjustment for washing; and (4)
adjustment for hand-to-mouth transfer.
Figure A.1 shows that non-dietary exposure pathway is estimated as a fraction of the
dermal pathway via hands using a coefficient representing the fraction of hands with residue
going into mouth at the ith day (i.e., HFi). Thus, the non-dietary exposure is subtracted from the
dermal exposure via hands. Final dermal exposure via hand at ith day (EdHand, i,final) is estimated
as:
)1(,,,, iwidHandfinalidHand HFEE −×= (A.6) Where,
EdHand, i, final = Final dermal exposure via hand [µg/kg]
EdHand,i,w = Dermal exposure via hands adjusted for washing effect [µg/kg]
HFi = Fraction of hands with residue going into mouth at the ith day [-]
×−=
aidHandi
aidHandwidHand EHW
EE
,,
,,,, )1(
(A.7)
Where, HWi = Hand washing removal efficiency at the ith day [-]
PWH = Probability of washing hands in a day [-]
EdHand,i,a = Adjusted dermal exposure via hands at the ith day [µg/kg]
),( ,,,, cidHandLHaidHand EEMinE
i= (A.8)
Where,
iLHE = Maximum dermal exposure via hand [µg/kg]
P<1-PWH (No Washing)
P>= 1-PWH (Washing)
142
B
handFractotaliLH W
SASALHE
i
××= (A.9)
LHi = Maximum dermal load via hand [µg/cm2]
SAhandFrac = Fraction of body surface area associated with hands [-]
SAtotal = Total body surface area [cm2]
54375.035129.0305 Btotal WHSA ××= (A.10)
H = Height [cm]
EdHand,i,c = Dermal exposure via hand corrected for the carryover effect
[µg/kg]
idHandidHandfcidHand EEEE ,1,,, +×= − (A.11) Where, Ef = Fraction of exposure from the prior day considered in the carryover
effect [-]
EdHand,i-1 = Dermal exposure via hands at the time step i-1 [µg/kg]
EdHand,i = Dermal exposure via hands at the time step i [µg/kg]
Where,
B
iettnonHandettnon
B
iettHandiettidHand W
EDTCSRW
EDTCSRE ,argarg,arg,arg
,−− ××
+××
= (A.12)
Where, SRtarget ,i = Residue at target surfaces at the ith day (measured or modeled)
[µg/cm2]
SRnon-target, i = Residue at non-target surfaces at the ith day [µg/cm2]
TChand = Hand dermal transfer coefficient measured [cm2/hr]
EDtarget,, i = Duration of exposure to target site at the ith day [hr]
EDtarget,, i = Duration of exposure to non-target site at the ith day [hr]
143
For surface residue at the ith day:
iRTRAett DFRSR )1(100arg −×××= (A.13-a)
iRTRTNAettnon DFRRSR )1(100 /arg −××××=− (A.13-b)
Where, RA = Application rate [g/m2]
RA = AreaAm (A.14)
Am = Mass of pesticide application [g]
Area = Applied area [m2]
RN/T = Non-target versus target ratio [-]
FTR = Fraction of chemical available for transfer [-]
DR = Fraction of residue that dissipates daily (decay rate) [1/day]
For the dermal exposure pathway via body the same equations as above exist. The dermal
exposure via body corrected for washing effect at the ith day is estimated as:
×−=
aidBodyi
aidBodywidBody EBW
EE
,,
,,,, )1(
(A.15)
Where, BWi = Body washing removal efficiency at the ith day [-]
PWB = Probability of washing body in a day [-]
EdBody,i,a = Adjusted dermal exposure via body at the ith day
),( ,,,, cidBodyLBaidBody EEMinEi
= (A.16) Where,
iLBE = Maximum dermal exposure via body [µg/kg]
B
BUhandFractotaliLB W
FSASALBE
i
×−××=
)1( (A.17)
LBi = Maximum dermal load via body [µg/cm2]
FBU = Fraction of body that is unclosed [-]
P<1-PWB (No Washing)
P>= 1-PWB (Washing)
144
EdBody,i,c = Dermal exposure via body corrected for the carryover
effect [µg/kg]
idBodyidBodyfcidBody EEEE ,1,,, +×= − (A.18) Where, EdBody,i-1 = Dermal exposure via body at the time step i-1 [µg/kg]
EdBody,i = Dermal exposure via body at the time step i [µg/kg]
B
iettnonBodyettnon
B
iettBodyiettidBody W
EDTCSRW
EDTCSRE ,argarg,arg,arg
,−− ××
+××
= (A.19)
Where, SRtarget ,t = Residue at target surfaces at the ith day (measured or
modeled) [µg/cm2]
SRnon-target, i = Residue at non-target surfaces at the ith day [µg/cm2]
TCBody = Body dermal transfer coefficient measured [cm2/hr]
EDtarget,, i = Duration of exposure to target site at the ith day [hr]
EDtarget,, i = Duration of exposure to non-target site at the ith day [hr]
Where the surface residue at the ith day is calculated using Equation (11). The average dermal
exposure from hand and body is estimated as:
m
EEE
m
iwidBody
m
ifinalidHand
Dermal
+
=∑∑
== 1,,
1,,
(A.20)
Non-dietary Ingestion Pathway
Based on the Alion suggestion, exposure via non-dietary ingestion pathway (EIngestion,i) is
a fraction of the dermal exposure via hands as:
)(,,, iwidHandiIngestion HFEE ×= (A.21)
The average exposure from non-dietary ingestion pathway is estimated as:
145
m
EE
m
iiIngestion
Ingestion
∑== 1
,
(A.22)
A.2 Probability Distributions of Model Inputs
The model input distribution assumptions are made mainly based upon SHEDS-Pesticide
model input parameters. Detailed information on the proposed model input assumptions is shown
in Table A.1. The information includes assumed probability distribution for each input,
corresponding parameters, and the sampling strategy.
146
Table A.1. Input Assumptions for the Simplified SHEDS model
Inputs Distribution Para_1a Para_2b Para_3c Unit Comment Sampling Strategy
Inhalation Pathway
Cair Triangular 0.1 1 4 [µg/m3] Concentration of applied pesticides monthly
Cb Lognormal 6.8×10-4 1.87 [µg/m3] Background air concentration
Cb∈[0, 0.85] monthly
Dinh,i Normal 550 140 [min] Inhalation exposure duration
Dinh, i∈[0, 1440] daily
PAI Normal 1.75 0.2 [-] Physical activity index
PAI∈[1, 4] daily
k Triangular 0.05 0.13 0.4 [1/day] Decay rate monthly
WB Lognormal 30 1.3 [Kg] Body weight
WB∈[16, 55] monthly
Dermal Pathway
AM Uniform 0.5 2.0 [g] Mass of Application monthly
Area Triangular 20 40 90 [m2] Applied Area monthly
BW Uniform 0.7 1.0 [-] Body washing removal efficiency daily
DR Triangular 0.1 0.2 0.4 [-] Fraction of residue that dissipates
dailymonthly
Ef Uniform 0.3 0.5 [-] Fraction of exposure from prior day daily
147
EDtarget Normal 6.0 1.4 [hr] Exposure duration at target area
EDtarget∈[0, 24] daily
EDnon-target Normal 7.5 1.6 [hr] Exposure duration at non-target area
EDnon-target∈[0, 24] daily
FTR Uniform 0.0 0.08 [-] Fraction of chemical available for
transfer daily
FBU Triangular 0.1 0.25 0.60 [-] Fraction of body that is unclosed daily
H Triangular 100 125 150 [cm] Height monthly
HW Uniform 0.3 0.9 [-] Hand washing removal efficiency daily
LH Uniform 0.1 0.7 [µg/cm2] Maximum dermal load via hand monthly
LB Uniform 0.1 0.7 [µg/cm2] Maximum dermal load via body monthly
PWB Triangular 0.3 1.0 1.0 [−] Probability of washing body monthly
PWH Triangular 0.3 1.0 1.0 [−] Probability of washing hands monthly
RN/T Uniform 0.05 0.15 [-] Non-target vs target ratio monthly
SAhandFrac Point 0.05 [-] Fraction of body surface area
associated with handsNA
TChand Lognormal 2029 1.3 [cm2/hr] Transfer coefficient (hand)
TChand∈[0, 6000] daily
TCbody Lognormal 5900 1.3 [cm2/hr] Transfer coefficient (body)
TCbody∈[0, 18000] daily
148
Non-Dietary Pathway
HF Triangular 0.025 0.05 0.25 [-] Fraction of Hands Going into Mouth
daily
Note: a. : For normal distribution, para_1 is the mean; for lognormal distribution, para_1 is geometric mean; for uniform and triangle distributions, the para_1 is minimum. b. : For normal distribution, para_2 is the standard deviation; for lognormal distribution, para_2 is geometric standard
deviation; for uniform distribution, para_2 is maximum, and for triangle distribution, the para_2 is peak. c. : For triangle distributions, the para_2 is maximum.
149
APPENDIX B
B.1 Algorithm for Application of Fourier Amplitude Sensitivity Test
Step 1: Assigning Frequencies to Inputs
• For time step t=1, assign incommensurate frequencies to inputs using the following
equation, where n represents the number of inputs and Ωn and dn are tabulated in Table
B-1:
+=Ω=
−+− inii
n
d 11
1
ωωω
(B-1)
• For example, for a case with four inputs the frequency set includes 5, 11, 19, and 23 for
ω1 to ω4, respectively.
• For time steps t=2,…,tn, where tn is the last day of the model simulation and for inputs
with daily sampling frequency generate a set of incommensurate frequencies using
Equation B-1 (See Matlab code provided in Figure B-1)
• Sample without replacement from generated set of frequencies for inputs with daily
sampling frequency (See Matlab code provided in Figure B-1) and assign these values to
daily inputs
Step 2: Setting the Search Variable
• Set the minimum sample size to be used in FAST as:
12 max +××= ωMN s (B-2)
where,
M = The interference factor (usually 4 or higher)
150
ωmax = The largest frequency among the set of ωi frequencies (including
frequencies for monthly and daily inputs) assigned to inputs in Step 1:
• Assign the sequence of sampled values for each input as:
s
sk N
Nks
′−′−
×=)12(
2π k = 1,2,…, '
sN (B-3)
where, 2
1' += s
sN
N
Step 3: Selecting Search Curves for Inputs
• Transform each input to a harmonic variable using:
)][sin( sGx iii ω= i = 1, 2, …, n (B-4)
where,
n = Number of inputs
ωi = Frequency assigned to input i in Step 3
Gi = Transformation function; )21)arcsin(1()( 1 +×= − uFuG
iPi π
F-1 = Inverse cumulative distribution function (ICDF) for the probability
distribution of inputs
• Figures B-2 and B-3 show Matlab codes for calculating the transformation functions for
selected probability distributions.
Step 4: Calculating the Main Effect Associated with Each Input
• Generate the total of 'sN Samples for each input d based on Step 3
• Feed sampled values to the model and calculate corresponding output values
• For the selected output, three alternative exposure scenarios can be considered:
o Daily exposure
151
o Incremental exposure:
>−===
−×××
=××
1,1,
)1(1
)(1
)(1
)1(1
)(1
tyyYtyY
tn
tn
tn
tn
tn
o Cumulative exposure:
∑=
×× =t
T
Tn
tn yY
1
)(1
)(1
where, n is the number of variability iterations (i.e., N’s), and yn×1(t) is selected output of
interest at time step t.
• Calculate Fourier coefficients Aj and Bj as:
[ ]
×++= ∑=
−+
0
)cos()()()(1'
1' 000
qN
jsYsYsYNA
s
N
qqNqNN
sj
q π (B-5)
[ ]
×−= ∑=
−+
0
)sin()()(1'
1' 00
qN
jsYsYNB
s
N
qqNqN
sj
q π (B-6)
Where, 2
1' −= s
qN
N and2
1'
0+
= sNN . In Equations (B-5) and (B-6),
Y(x1(s),x2(s),…,xn(s)) is shown as Y(s) for simplicity.
• Calculate the output variance as:
∑∞
=
+=1
222 )(2j
jj BAσ (B-7)
• Figures B-4 and B-5 show Matlab codes for estimating the output variance and Fourier
coefficients for different inputs, respectively
• Estimate the partial variance associated with the ith input as:
If j even If j odd
If j odd If j even
152
∑=
+=max
1
222 )(2p
ppp iii
BA ωωωσ (B-8)
• Estimate the main effect associated with input xi as:
2
2
σσ ω
ωi
iS = (B-9)
Step 5: Calculating the Total Effect Associated with Each Input
• In order to estimate the total sensitivity index for input xi, assign a new set of frequencies
to inputs based on the code provided in Figure B-6.
• Generate the total of 'sN Samples for each input d based on Step 4
• Feed sampled values to the model and calculate corresponding output values
• For the selected output, three alternative exposure scenarios can be considered:
o Daily exposure
o Incremental exposure:
>−===
−×××
=××
1,1,
)1(1
)(1
)(1
)1(1
)(1
tyyYtyY
tn
tn
tn
tn
tn
o Cumulative exposure:
∑=
×× =t
T
Tn
tn yY
1
)(1
)(1
where, n is the number of variability iterations (i.e., N’s), and yn×1(t) is selected output of
interest at time step t.
• Estimate the output variance using the code provided in Figure B-4
• Add all the spectral components in the frequency range [1, ωi/2], where the spectrum of
the Fourier series expansion is defined as 22jjj BA +=Λ . The summation of all the
153
spectral components in that frequency range provides the value of 2)( i−ωσ that is the
portion of the output variance arising from the uncertainty of all inputs except xi.
• Estimate the total sensitivity index for input xi as:
2
2)(1
σσ ω
ωi
iST −−= (B-10)
where, 2σ is the output variance based on the new frequencies.
154
Table B-1. Parameters Used in Calculating Frequency Sets Free of Interference to Fourth Order N Ωn dn N Ωn dn
1 0 4 26 385 416 2 3 8 27 157 106 3 1 6 28 215 208 4 5 10 29 449 328 5 11 20 30 163 198 6 1 22 31 337 382 7 17 32 32 253 88 8 23 40 33 375 348 9 19 38 34 441 186 10 25 26 35 673 140 11 41 56 36 773 170 12 31 62 37 875 284 13 23 46 38 873 568 14 87 76 39 587 302 15 67 96 40 849 438 16 73 60 41 623 410 17 85 86 42 637 248 18 143 126 43 891 448 19 149 134 44 943 388 20 99 112 45 1171 596 21 119 92 46 1225 216 22 237 128 47 1335 100 23 267 154 48 1725 488 24 283 196 49 1663 166 25 151 34 50 2019 0
155
Figure B-1: Matlab Code for Assigning Frequencies to Daily Inputs.
156
Figure B-2: Matlab Code for Selecting a Search Curve for Inputs with Uniform, Loguniform, and Normal Distributions.
157
Figure B-3: Matlab Code for Selecting a Search Curve for Inputs with Lognormal Distribution.
158
Figure B-4: Matlab Code for Estimating the Output Variance.
159
Figure B-5: Matlab Code for Estimating the Fourier Coefficients.
160
Figure B-6: Matlab Code for Assigning New Frequencies to Daily Inputs Used in Estimation of
Total Sensitivity Indices.
161
B.2 Algorithm for Application of Sobol’s Method
Figure B-7 shows an algorithm for application of Sobol’s method. Each step is explained in the
following.
Step 1: Generating Two Sets of Random Samples from Probability Distributions of Inputs
• Two set of random samples from probability distributions of inputs should be generated
and stored (A1, B1, A’1, and B’2, where A and B are matrices that hold the generated
values for monthly and daily inputs, respectively).
Step 2: Running the Model for Sampled Values of Inputs
• Run the model based on random samples generated in Step 1 and estimate values of
selected outputs and store the output values in a matrix format (y and y’, where y is the
matrix that holds the output values).
• For the selected output, three alternative exposure scenarios can be considered:
o Daily exposure
o Incremental exposure:
>−===
−×××
=××
1,1,
)1(1
)(1
)(1
)1(1
)(1
tyyYtyY
tn
tn
tn
tn
tn
o Cumulative exposure:
∑=
×× =t
T
Tn
tn yY
1
)(1
)(1
where, n is the number of variability iterations, and yn×1(t) is selected output of interest at time
step t.
162
Step 3: Forming New Matrices Required for Sensitivity Indices
• Form matrix Mj and Nj for input xj based on the values stored in matrices A1, B1, A’1,
and B’2 in Step 1 (see Figures B-8 and B-9)
• Figure B-10 shows a Matlab code provided for forming Mj and Nj matrices. For this
example, inputs have monthly and daily resampling frequencies.
Step 4: Estimating the Sensitivity Indices and Performing Bootstrap Simulation to
Calculated the Confidence Intervals
• Run the model based on the matrices formed in Step 3 and save the output values (YM
and YN for output values based on M and N, respectively).
• For the selected output, three alternative exposure scenarios can be considered:
o Daily exposure
o Incremental exposure:
>−===
−×××
=××
1,1,
)1(1
)(1
)(1
)1(1
)(1
tyyYtyY
tn
tn
tn
tn
tn
o Cumulative exposure:
∑=
×× =t
T
Tn
tn yY
1
)(1
)(1
where, n is the number of variability iterations, and yn×1(t) is selected output of interest at time
step t.
• Start the bootstrap simulation
o Suggested number of bootstrap simulations is 10,000 based on Archer et al. (1997)
o At each bootstrap simulation randomly sample the “output” values with replacement
and form the following new output matrices:
163
BY : Output matrix based on bootstrap sampling of Y from Step 2
'BY : Output matrix based on bootstrap sampling of Y’ from Step 2
MBY : Output matrix based on bootstrap sampling of YM
NBY : Output matrix based on bootstrap sampling of YN
o The Key point in the bootstrap sampling step is to use the same set of random
numbers when generating BY , 'BY , M
BY , and NBY .
o Main effect for input xj can be estimated using the following equation:
)())(( 2
yVyEU
S jj
−= (B-11)
where,
E2(y) = Square of expected value for the selected output:
∑=
′′′××=n
rrkrrrkrr xxxfxxxf
nyE
12121
2 ),,,(),,,(1)( LL (B-11-1)
V(y) = Variance of the selected output estimated based on BY
Uj = ∑=
+− ′′′′′××−
n
rrkjrrjjrrrrkrr xxxxxxfxxxf
n 1)1()1(2121 ),,,,,,,(),,,(
11
LLL
(B-11-2)
f(.) = Mean model output over the selected time period (e.g., one
month) for rth individual; r = 1,…, n
n = Number of variability iterations
x = Sampled values of inputs from Step 1
x’ = Resample values of inputs from Step 3
1 2
1 3
164
(1) = BY
(2) = 'BY
(3) = MBY
o Total effect for input xi can be estimated using the following equation:
)())((
12
yVyETU
S jTj
−−= − (B-12)
where,
ET2(y) = Square of expected value for the selected output:
2
121
2 ),,,(1)(
×= ∑=
n
rrkrr xxxf
nyET L (B-12-1)
U-j = ∑=
+− ′××−
n
rrkjrrjjrrrrkrr xxxxxxfxxxf
n 1)1()1(2121 ),,,,,,,(),,,(
11
LLL
(B-12-1)
f(.) = Mean model output over the selected time period (e.g., one
month) for rth individual; r = 1,…, n
n = Number of variability iterations
x = Sampled values of inputs from Step 1
x’ = Resample values of inputs from Step 3
(1) = BY
(4) = NBY
1 4
1
165
Step 5: Summarizing the Results
• Step 4 should be repeated for time steps t=1 to 30 (for one month simulation time)
• Estimate the average main and total effects at each day
• Estimate the 95% confidence intervals for the main and total effects at each day
166
Figure B-7. Flow Diagram for Application of the Sobol’s Method.
167
Figure B-8. Forming Mj and Nj Matrices for Input xj with Monthly Sampling Frequency.
168
Figure B-9. Forming Mj and Nj Matrices for Input xj with Daily Sampling Frequency.
169
Figure B-10. Matlab Code for Forming M and N Matrices for Inputs with Monthly and Daily
Sampling Frequencies.