claude beigel, phd. exposure assessment senior scientist research triangle park, usa practical...
TRANSCRIPT
Claude Beigel, PhD.
Exposure Assessment Senior Scientist
Research Triangle Park, USA
Practical session metabolitesPart II: goodness of fit and decision making
2
Recommended Tools for Assessing Goodness of Fit
Same recommended approach as for parent substance
Combination of visual assessment and statistical tests
Visual assessment, although bringing some level of subjectivity, is necessary to discern between normal data variability (scattering) and systematic model deviation, this is not done by the statistical tests
3
Recommended Tools for Assessing Goodness of FitVisual Assessment
Visual check of model description of measured data and distribution of residuals (plot of residuals, Predicted - Observed)
Systematic deviation indicates kinetic model may not be appropriate (unless deviation can be attributed to experimental artifacts)
Residual Plot Metabolite
-20
-15
-10
-5
0
5
10
15
20
0 50 100 150 200 250 300 350 400 450 500
Time (days)
Re
sid
ua
l (%
AR
)
ParentMetabolite
Parent SFO, metabolite SFO
0 50 100 150 200 250 300 350 400 450 500
Time (days)
0
10
20
30
40
50
60
70
80
90
100
Su
bs
tan
ce (
% A
R)
Example 8.2 of report, SFO-SFO fit
4
Recommended Tools for Assessing Goodness of FitVisual Assessment
Residuals should be randomly distributed on vertical axis
ParentMetabolite
Parent SFO, metabolite FOMC
0 50 100 150 200 250 300 350 400 450 500
Time (days)
0
10
20
30
40
50
60
70
80
90
100
Su
bs
tan
ce (
% A
R)
Residual Plot Metabolite
-20
-15
-10
-5
0
5
10
15
20
0 50 100 150 200 250 300 350 400 450 500
Time (days)
Re
sid
ua
l (%
AR
)
Example 8.2 of report, SFO-FOMC fit
5
Recommended Tools for Assessing Goodness of FitStatistical Indices
Chi2 (2) statistical test
Minimum error percentage to pass 2 test at a 5% significance level
– Needs to be performed for each substance individually, to avoid that good fit of the main substances (parent and/or major metabolite) overshadows goodness of fit of more minor substances (weighting issue)
– Calculated from fitted versus observed substance data (use of average values for replicates is recommended)
– Degrees of freedom for the substance defined as number of substance data points used in 2 test minus number of estimated parameters for the substance
Do not count replicates if averages used, excludes data points set to 0 (metabolite at time-0) or not counted (<LOD/LOQ)
Metabolite parameters defined as metabolite formation fraction and degradation rate parameters (dependent of kinetic model used)
6
Recommended Tools for Assessing Goodness of FitStatistical Indices
One-sided t-test for evaluating uncertainty of rate constant parameters
To determine whether rate is significantly different from 0
– If p < 0.05, parameter is considered significantly different than zero
– If p between 0.05 and 0.1, weight of evidence should be considered
Especially important for metabolites that do not show a clear decline
Because parameters in parent + metabolite fits (formation and degradation parameters) can be highly correlated, the t-test is performed at final step (all parameters fitted together)
– Degrees of freedom defined as number of data points (including replicates) minus number of fitted parameters
7
Recommended Tools for Assessing Goodness of FitData Handling / Methodology
Basic data handling
Paste ModelMaker output (integration table) in Excel spreadsheet
– Extract fitted values corresponding to measured times for each substance
– Average replicates if necessary
(an automated Excel spreadsheet may be created for that purpose, but not available yet)
Minimum 2 error % for metabolites may be calculated using Parent degradation kinetics.xls file
Paste measured Vs. fitted values in Chi2 all models worksheet, update number of parameters cell and click calculate
8
Recommended Tools for Assessing Goodness of FitData Handling / Methodology
Residuals may be plotted in Parent degradation kinetics.xls file
Valid only for 1- or 2-replicate data sets (if more, needs to be done manually)
Paste measured Vs. fitted values (all replicates) in SFO no-reps or SFO 2-reps worksheet
t-test for rate constant parameters may be performed using provided t-test.xls file
For each rate constant parameter, enter parameter estimate and standard error, number of data points and number of parameters estimated
9
Hands-on Example 1
Exercise 1
Open ModelMaker file for example 1
From result table, extract fitted value for each sampling time, write down in output tables for parent, metabolite1 and metabolite2
Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically
10
Hands-on Example 1
Visual assessment
Check ModelMaker plot of fit and answer following questions for each substance
– Does fitted line adequately describe data, are there obvious over- or under-predictions (including day-0)
Plot residuals for each substance and answer following questions
– Do residuals show distinct pattern, are most of the points above or below 0-line, what is the magnitude?
Statistical indices
Calculate minimum 2 error percentage for each substance
Perform t-test for all rate constant parameters (parent and metabolites) and record P-value and conclusion
11
Hands-on Example 1Visual Assessment
Graph Assessment / Remarks
ParentOverall fit
Residuals
Metabolite1
Overall fit
Residuals
Metabolite2 Overall fit
Residuals
12
Hands-on Example 1Statistical Indices
2-test Relevant Parameters
Estimated (y/n)Number of Parameters
Minimum 2 Error
Percentage
ParentPini
kP
Metabolite1ffM1
kM1
Metabolite2ffM2
kM2
t-test Estimated Value
Standard Error
Number of Data Points
Number of Estimated
ParametersP-value Conclusion
kP
kM1
kM2
13
Hands-on Example 2, Parent FOMC
Exercise 2
Open ModelMaker file for example 2, parent FOMC
From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite
Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically
Perform visual assessment and calculate statistical indices
14
Hands-on Example 2, parent FOMCVisual Assessment
Graph Assessment / Remarks
ParentOverall fit
Residuals
Metabolite
Overall fit
Residuals
15
Hands-on Example 2, parent FOMCStatistical Indices
2-test Relevant Parameters
Estimated (y/n)Number of Parameters
Minimum 2 Error
Percentage
Parent
Pini
P
P
MetaboliteffM
kM
t-test Estimated Value
Standard Error
Number of Data Points
Number of Estimated
ParametersP-value Conclusion
kM
16
Hands-on Example 2, parent DFOP
Exercise 3
Open ModelMaker file for example 2, parent DFOP
From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite
Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically
Perform visual assessment and calculate statistical indices
17
Hands-on Example 2, parent DFOPVisual Assessment
Graph Assessment / Remarks
ParentOverall fit
Residuals
Metabolite
Overall fit
Residuals
18
Hands-on Example 2, parent DFOPStatistical Indices
2-test Relevant Parameters
Estimated (y/n)Number of Parameters
Minimum 2 Error
Percentage
Parent
Pini
g
k1
k2
MetaboliteffM
kM
t-test Estimated Value
Standard Error
Number of Data Points
Number of Estimated
ParametersP-value Conclusion
k1
k2
kM
19
Hands-on Example 2, Metabolite Decline
Exercise 4
Open ModelMaker file for example 2, metabolite decline
From result table, extract fitted value for each sampling time, write down in output tables for parent and metabolite
Enter values in Metabolitesexamplesoutput.xls, averages are calculated automatically
Perform visual assessment and calculate statistical indices
20
Hands-on Example 2, Metabolite DeclineVisual & Statistical Assessment
Graph Assessment / Remarks
Metabolite Decline
Overall fit
Residuals
2-test Relevant Parameters
Estimated (y/n)Number of Parameters
Minimum 2 Error
Percentage
MetaboliteMmax
kM
t-test Estimated Value
Standard Error
Number of Data Points
Number of Estimated
ParametersP-value Conclusion
kM
21
Decision MakingTrigger Endpoints
Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with best-fit model for parent)?
Yes use SFO DT50/90 endpoints
No and clear decline of metabolite, use FOMC model for metabolite (in combination with best-fit model for parent)
– If FOMC acceptable based on visual assessment and statistical indices, use FOMC DT50/90 endpoints
– If not, model decline of metabolite with best-fit model and use decline DT50/90 as conservative endpoints
No and no apparent decline of metabolite
– Assess relevance of study with regard to metabolite
– Check other studies
– Study with metabolite may be needed
22
Decision MakingModeling Endpoints
Based on visual assessment and statistical indices, is SFO model acceptable for the metabolite (in combination with appropriate model for parent)?
Yes use modeling endpoints for metabolite
No and clear decline
– If formation fraction estimate is reliable, use with decline rate constant as conservative endpoints
– If not, use formation fraction of 1 with decline rate constant as conservative endpoints
– If metabolite biphasic, use appropriate higher-Tier approach (e.g. DFOP, PEARL)
– If terminal metabolite and biphasic, use FOMC DT90/3.32 as half-life
No and no apparent decline of metabolite
– Assess relevance of study with regard to metabolite
– Check other studies
– Study with metabolite may be needed
23
Hands-on Examples
Determine appropriate trigger and modeling endpoints for Example 1 metabolites 1 and 2 and Example 2 metabolite