A new exact test to globally assess a population PK and/or PD model

C. Laffont & D. Concordet

UMR 181 Physiopathologie et Toxicologie ExpérimentalesINRA, ENVT

ECOLENATIONALEVETERINAIRET O U L O U S E

• Weighted residuals

– WRES NONMEM

– CWRES Hooker et al. (2007) Pharm Res 24:2187

– PWRES Monolix

Classical metrics for global model evaluation

Weighted residuals: difficulty in interpretation

Simulated data analysed with the correct model !Simulated data analysed with the correct model !

CWRES WRES

Karlsson & Savic, 2007,Clin Pharmacol Ther 82:17

Diagnosing Model Diagnostics

Weighted residuals (WR) calculation:

)()( 2/1iiii YEYYVarWR

• Calculated from the vector of observations Yi in subject i and denoted here WRi

• Simulation (PWRES)

• FO/FOCE approximation of the model (WRES,CWRES)

• Simulation (PWRES)

• FO/FOCE approximation of the model (WRES,CWRES)

Expected distribution if the model is correct ?

I)N(0,~WRi

For linear Gaussian models, Yi distribution is Gaussian so:

For nonlinear models, WRi distribution is unknown !!!

Recently proposed metric: NPDE

• Simulation-based approach

– compares at each time j the distribution of WRij with their predictive distribution according to the model

Brendel et al. (2006) Pharm Res 23: 2036

Expected distribution if the model is correct ?

NPDE are assumed to be independent and Gaussian:

I)N(0,~NPDE

Independence issue rightly discussed by the authors:

when NPDE are dependent, they are jointly not Gaussian

New metric : GUD (Global Uniform Distance)

The purpose of this work was to propose:

• an exact test for global model evaluation

• an easy diagnostic graph with no subjective interpretation

Reject model with 5% risk= black line (data) outside the ring = black line (data) inside the ring

Do not reject model

GUD calculation & testing

• Step 1Step 1

– As for calculation of NPDE, we compute for each subject i the vector of WRi

)(ˆ)(ˆ 2/1iiii YEYYarVWR

M (=2000) simulations (unbiased)

2/1)( iYVar is the Cholesky decomposition of the full variance matrix of Yi

WRi are decorrelated within subject i

Decorrelation does not imply independence !!!

Decorrelation= independence

Decorrelation= independence

Decorrelation independence

Decorrelation independence

• Only true for linear Gaussian models

iiiii tY 21

• Case of nonlinear models

1 compt i.v. model

We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere

WRi vector (observations)

Unit sphere

r = 1… R

• Step 2:Step 2:– To handle data dependency, we use a recent random

projection method (See Cuesta-Albertos et al. (2007) for an application)

vector ei1

ProjectionProjectioni1i1

GUD calculation & testing

Unit sphere

WRi vector (observations)

vector ei2ProjectionProjectioni2i2

GUD calculation & testing

• Step 2:Step 2:– To handle data dependency, we use a recent random

projection method (See Cuesta-Albertos et al. (2007) for an application)

r = 1… R

We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere

Each subject i

)(, WRiiri feWRProjection on R (=100) random directions

...

)(WRif

Subject i

)(1 WRf

Subject 1

GUD calculation & testing

cdf cdf

)(2 WRf

Subject 2

Random pdf independent between subjectsRandom pdf independent between subjects

Mixture of projection distributions

GUD calculation & testing

• Step 3:Step 3:

– Compare this global cdf obtained for the sample to its distribution under H0 (i.e. correct model)

cdf cdf

SampleSample Simulations under HSimulations under H00

95% prediction region

95% prediction region

For each replicate, we compute the maximal absolute distance

from mean cdf curve (GUD)

GUDGUD

= Global Uniform Distance= Global Uniform Distance

00

K cdf curves

Calculation of 95% prediction region under H0

K =(5000) replicates of the study design

Simulations under H0 with tested model

Simulations under H0 with tested model

mean cdfmean cdf

95%5%

5% of curves that 5% of curves that are the most distant are the most distant

from mean cdffrom mean cdf

5% of curves that 5% of curves that are the most distant are the most distant

from mean cdffrom mean cdf

Calculation of 95% prediction region under H0

mean cdfmean cdf

Uniform region containing 95% of curves

Uniform region containing 95% of curves

= Global Uniform Distance= Global Uniform Distance

95% prediction region

For each replicate, we compute the maximal absolute distance

from mean cdf curve (GUD)

K cdf curves

K =(5000) replicates of the study design

Simulations under H0 with tested model

Simulations under H0 with tested model

GUDGUD0

95%

0

GUDGUD

GUD test for your sample

Sample cdf

Sample

P valueP value

Do not reject modelDo not reject modelSimulations under H0Simulations under H0 True modelTrue model

= Global Uniform Distance= Global Uniform Distance

5%

95% prediction region

0

5%

P valueP value

Sample

Wrong modelWrong model

Sample cdf

GUD test for your sample

Reject modelReject model

GUDGUD

= Global Uniform Distance= Global Uniform Distance

95% prediction region

Sample

QQ ring diagnostic plot

Reject modelReject model

Sample

Do not reject model

Do not reject model

PK model PK/PD model Linear model

1 compt i.v. bolus Sigmoidal Emax model (γ=4)

Exponential IIV on CL, V

Proportional res. error

Exponential IIV on Emax, EC50

Additive res. error

Additive IIV on 1, 2

Additive res. error

Performances under H0: GUD vs. other metrics

iiiii tY 21

• Simulations under H0 to evaluate the level of the tests 100 subjects i.i.d with 4 obs./ subject 5000 replications of study design

• Weighted residuals & NPDE calculated in C++ Kolmogorov-Smirnov (KS) test to test for a N(0,1)

PK model

PK/PD model

Linear model

KS testfor N(0,1)

WRES

CWRES

PWRES

NPDE

100

57

66

4.0

100

13

62

7.5

4.9

4.9

4.5

4.6

GUD test GUD 5.1 5.0 4.9

Level of the tests under H0

Type I error (%) - nominal level = 5%

95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)

p value

The p-value should follow a uniform distribution under H0 !

Exp

ecte

d v

alu

e fr

om

u

nif

orm

dis

trib

uti

on

05.0)05.0(:Ex valuepProb

PP plotPP plot

Level of the tests under H0

GUD testGUD test

PK model PK/PD model

p value p value

Exp

ecte

d v

alu

e fr

om

u

nif

orm

dis

trib

uti

on

95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)

Good whateverGood whatever the level !the level !

Good whateverGood whatever the level !the level !

Level of the tests under H0

NPDE: KS test for N(0,1) NPDE: KS test for N(0,1)

PK model

95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)

p value p value

Exp

ecte

d v

alu

e fr

om

u

nif

orm

dis

trib

uti

on

or or of of type I errortype I error or or of of type I errortype I error

PK/PD model

Level of the tests under H0

Conclusion• Poor performances of weighed residuals

• NPDE show much better performances but do not deal with the issue of data dependency within subjects

– Possible increase or decrease of type I error depending on model

• New test and graph based on GUD metric

– Encouraging results

– More work needed to evaluate this test under more complex conditions (different sampling times per subject, real-case data…)