disk diffusion breakpoint determination for antimicrobial

Disk Diffusion Breakpoint Determinationfor Antimicrobial Susceptibility Testing

Glen DePalma

Joint work with Professor Bruce A. Craig

Department of Statistics

March 20, 2012

Glen DePalma Breakpoint Determination for Susceptibility Testing 1 / 46

Dilbert!


Background

I Misuse of antibiotics is a major contributor to the growth ofdrug-resistant bacteria.

I Two clinical tests assess susceptibility of bacterial pathogen to drug

I Drug Dilution (MIC) → concentration (µg/ml)

I Disk Diffusion (DIA) → zone diameter (mm)

I Drug/Test breakpoints classify pathogen to be:

I Susceptible - very high likelihood of successful treatment

I Intermediate - marginal likelihood of successful treatment

I Resistant - low likelihood using FDA approved dosage


MIC Test

www.AntimicrobialTestLabs.com


DIA Test


Properties of the Two Tests

I Drug Dilution (MIC) test

I Considers set of two-fold dilutions

I Will use log2(MIC) = . . . ,−4,−3,−2,−1, 0, 1, 2, 3, 4, . . .

I Observed MIC: lowest dilution with no visible growth

I Test rounds up to nearest dilution (integer)

I Disk Diffusion (DIA) test

I Observed DIA: diameter of clear zone surrounding disk

I Rounded to the nearest mm

I Inverse relationship between true MIC and true DIA


Background

I Both MIC and DIA breakpoints are set by agencies such as theClinical and Laboratory Standards Institute (CLSI) and the Food andDrug Administration (FDA).

I MIC breakpoints (log2 concentrations) based primarily on thepharmacokinetics and pharmacodynamics of the drug.

I Given the MIC breakpoints, the DIA breakpoints must be estimated.

I Relationship between DIA and MIC is drug/pathogen specific withunknown functional form.

I DIA is more often used in hospitals because it is faster and requiresless lab space.


Problem

How to best determine appropriate DIA breakpoints?


Determination of DIA Breakpoints - Current Practice

I The error rate bounded (ERB) method is currently used by agenciesto determine DIA breakpoints.

I Perform MIC & DIA tests on hundreds of strains of a particularpathogen and then find the DIA breakpoints that minimize anobserved classification discrepancy index.

I Discrepancies based on Very Major (VM), Major (M), and minor (m)errors

I Very Major: MIC test = R and DIA test = S

I Major: MIC test = S and DIA test = R

I Minor: One test = I


Example MIC DIA Test Data


Error Rate Bounded Method


Concerns with ERB Method

I Simulation studies (Craig, 2000) show the ERB method:

I Is biased

I Lacks precision

I Focuses on observed test results

I Does not take into account MIC test characteristics:

I Rounding Up

I 3-fold variability


3-fold Variability

Table 1. Quality Control Study - MIC test: 10 Labs, 50 reps/labE. coli ATCC 25922

MIC LAB All1 2 3 4 5 6 7 8 9 10 Labs

-8 8 6 7 2 7 1 31-7 36 41 32 48 48 33 41 15 33 35 362-6 6 3 11 2 17 2 35 16 14 106-5 1 1

Mean -7.54 -7.57 -7.42 -7.21 -7.79 -7.06 -7.62 -6.93 -7.12 -7.22 -7.34StDev 0.46 0.37 0.52 0.12 0.12 0.14 0.35 0.14 0.42 0.38 0.43

Example of the rounding up effect on MIC data


MIC Model

I Distinguish observed test result, xi , from true result, mi

I To generate observed pair:

xi = dmi + εie ε ∼ N (0, σ2m)

I Given MIC breakpoints ML and MU one can calculate the probabilityof correct classification:

pMIC (m) =

Pr(x ≤ ML) = Φ

(ML−mσm

)m ≤ ML

Pr(ML < x < MU) = Φ(

MU−1−mσm

)− Φ

(ML−mσm

)ML < m < MU

Pr(x ≥ MU) = 1− Φ(

MU−1−mσm

)m ≥ MU


Probability of Correct MIC Classification


New Method Needed

I We propose an underlying model of the process that focuses on theprobability of correct classification.

I We develop a hierarchical model that uses latent MIC test means tolink the two observed results as well as describe the probability ofcorrect classification for each test.

I Our model accounts for the inherent test variabilities and differingtesting properties by linking the observed test pairs to a latent 1-1function of ’true’ test results.

I This should result in better precision and increased accuracy.


Model Based Approach

I The observed scatterplot of test results are used to estimate themodel parameters using Bayesian inference.

I Use model parameters to determine DIA breakpoints with best’performance’.

I More intuitive in the fact that the performance curves do not directlydepend on the observed results and also provides a visual descriptionof each test rather than a joint numerical summary.


Model Based Approach

Summarized in four steps:

1. Observed Data Given True Test Value

2. True Pathogen MIC Distribution

3. Relationship between True Test Values

4. Calibration based on Probability of Correct Identification


Observed Data Given True Test Value

I Distinguish observed test pair from true pair

I Observed pairs: xi and yi

I True pairs: mi and di

I Observed data are integers due to rounding

I To generate observed pair:

xi = dmi + εie yi = [di + δi ]

ε ∼ N (0, σ2m) δ ∼ N (0, σ2

d)


True Pathogen MIC Distribution

I Describe underlying pathogen MIC distribution as a mixture ofNormals:

π(mi |k ,µ,σ,p) =k∑

j=1

pj N (mi ;µj , σj)


Relationship between True Test Values

I Relationship, di = f (mi ), has been assumed to be linear (Rudrik etal., 1985) or logistic (Craig, 2000).

I Here we propose a more flexible nonparametric approach usingI-splines (Ramsey, 1988) which restricts the relationship to bemonotonic.

Assume a = t0 < ... < tq = b is a partition of [a, b] and y1, ..., y2k+q−1 is the knotsequence of the M-spline of order k . Then the i th I-spline

Ii (x ; yi , yi+1, ..., yi+k) =

∫ x

a

Bi (t)dt

Where Bi (t) is the i th M-spline basis function


Example Splines

M-splines - knot locations: (0, .2, .4, .6, .8, 1)

Resulting I-splines


Example of Hierarchical Model


Probability of Correct Identification

I Probability model links observed MIC results to true MIC

I Can determine probability of correct identification

I Assume true MIC breakpoints same as test breakpoints

I Given breakpoints ML, MU , DL, and DU

pDIA(m) =

Pr(y ≥ DU) = 1− Φ

(DU−.5−d

σd

)m ≤ ML

Pr(DL < y < DU) = Φ(

DU−.5−dσd

)− Φ

(DL+.5−d

σd

)ML < m < MU

Pr(y ≤ DU) = Φ(

DL+.5−dσd

)m ≥ MU


Calibration

I ’Calibrate’ the two probability of correct identification curves

I Use loss function:

L =

∫ ∞−∞

min (0, pDIA(u)− pMIC(u))2 w(u)du


Probability of Correct Identification Curves

Given the MIC probability of correct identification curve (fixed), presented is the DIAprobability of correct identification curve for a particular DU and DL.


Bayesian Inference

I Interested in the joint posterior of model parameters:

θ = {I1...q, k ,µ,σ,p}

π(θ|{x, y}) ∝ π({x, y}|{m,d},θ)π({m,d}|θ)π(θ)

I Given fixed knot sequence, use Markov Chain Monte Carlo (MCMC)to estimate parameters:

1. True test values:

I m, d

2. Underlying mixture of Normal parameters:

I µ,σ, p

3. Spline Coefficients:

I I1...q


MCMC

I Use when posterior is analytically intractable

I Construct a Markov chain whose stationary distribution is the targetdistribution

I Metropolis et al (1953) showed how this can be done

I Method generalized by Hastings (1970)


Metropolis Hastings

I Suppose the specified distribution has unnormalized density h. TheMetropolis-Hastings update does the following:

1. When the current state is x , propose a move to y , having conditional probability densitygiven x denoted q(x , .)

2. Calculate the ratio:

r(x , y) =h(y)q(y |x)

h(x)q(x |y)

3. Accept the proposed move y with probability:

a(x , y) = min(1, r(x , y))

I So the state after the update is y with probability a(x , y) and x withprobability 1− a(x , y)


Likelihoods

I Use MH proposals to update each of the parameters during each iteration

I π({x, y}|{m, d},θ):

n∏i=1

1√2πσ2

m

exp

(−(xi − mi )

2

2σ2m

)−

1√2πσ2

m

exp

(−(xi − 1− mi )

2

2σ2m

)+

1√2πσ2

d

exp

(−(yi + .5− di )

2

2σ2d

)−

1√2πσ2

d

exp

(−(yi − .5− di )

2

2σ2d

)

I π({m, d}|θ):

n∏i=1

k∑j=1

pj1√

2πσ2j

exp

−(mi − µj )2

2σ2j


Priors π(θ)

I µ,σ,p

I µ - ordered: k!(

1µu−µl

)kI 1

σ2 ∼ Gamma(3, 1)

I p ∼ Dirichlet(1, 1, ..., 1)

I log(I1...q)

I N (0, σ = 10)


MCMC Process for Our Model

1. Update mI Proposal: N (m, .5)

2. Update µ,σ,p based on new m

I Propose new neighboring Normal components for m

I Update p

I Propose new µ′i s (proposal uniform between µi−1 and µi+1)

I Update σ

3. Propose new or collapse two Normal components (RJMCMC)

4. Update Spline Coefficients (I)I Proposal: Adaptive multivariate normal: In+1 ∼ exp{N (log(In),Σ ∗ s)}

5. Update dI Deterministic update given m and In+1

6. Update number and location of knots - work in progress


RJMCMC

I Can use RJMCMC to dynmatically change number of normalcomponents

I Provides framework for MCMC in which the dimension of theparameter space can vary between iterations

I Applications: change-point models, finite mixture models, variableselection, ...

I Change from model (k , θk) to (k ′, θ′k) with probability:

min

(1,π(k ′,θ′

k)|x)q(k ′ → k)qdk′→k(u′)

π(k ,θk)|x)q(k → k ′)qdk→k′ (u)

∣∣∣∣δgk→k′ (θk ,u)

δ(θk ,u)

∣∣∣∣)


RJMCMC

I Dimensions and first and second moments of each model must match

I Often must introduce random variables

I Consider mixture of normals. Combine two mixtures:

pj∗ = pj1 + pj2

pj∗µj∗ = pj1µj1 + pj2µj2

pj∗(µ2j∗ + σ2

j∗) = pj1(µ2j1 + σ2

j1) + pj2(µ2j2 + σ2

j2)

I Similar idea for splitting one component into two but must introducethree random variables for dimension matching.


Simulation

I Used fixed set of model parameters

I Generated 300 scatterplots with 500 pathogens

I Computed DIA breakpoints for each scatterplot

I Summarize distribution of ’best’ DIA breakpoints


Simulation 1 - Linear Relationship

Example Data


Simulation 1 - Linear Relationship Results

Our Approach ERB


Simulation 2 - Logistic Relationship

Example Data


Simulation 2 - Logistic Relationship Results

Our Approach ERB


Simulation 3 - Convex Relationship

Example Data


Simulation 3 - Convex Relationship Results

Our Approach ERB


Conclusions

I Because of the increasing number of moderately susceptible andresistant isolates, choosing appropriate breakpoints has become moreof a statistical problem.

I The ERB method, while simple to implement, is too sensitive to thevariability in the observed frequencies caused both by the inherentexperimental error in each test and the choice/location of the isolatesrelative to the intermediate zone.

I Our model approach has shown increased accuracy and betterprecision.

I Working with subgroup of CLSI for making method available toclinicians.


Additional Ongoing Work

I Real data sets often have censored values:

I ex. MIC ≥ 5 instead of MIC = 5

I Difficult to deal with in a nonparametric setting


Additional Ongoing Work

I Assumed the measurement errors for MIC and DIA are known

I Reasonable given the abundance of past data

I However would like to estimate these parameters as well

I We have presented this approach to CLSI (Nov 2011)I Interest shown in new approach

I Actively developing software


Acknowledgments

I Purdue Department of Statistics

I Professor Bruce A. Craig

I Professor John Turnidge (Chair: Antimicrobial Testing Sub-Committee)

I Professor Laura Sands (Department of Nursing)I Support from grant from the National Institutes of Health (R01AG034160)

References:

I Brooks, Steve. Handbook of Markov Chain Monte Carlo. Boca Raton: CRC/Taylor & Francis, 2011.

I Craig, Bruce A. ”Modeling Approach to Diameter Breakpoint Determination.” Diagnostic Microbiology and InfectiousDisease 36.3 (2000): 193-202.

I Green, P.J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination.Biometrika 82, 711-732.

I Ramsay, J.O. ”Monotone Regression Splines in Action.” Statistical Science 3.4 (1988) 425-41.

I Turnidge, J., and D.L. Paterson. ”Setting and Revising Antibacterial Susceptibility Breakpoints.” Clinical MicrobiologyReviews 20.3 (2007): 391-408.


disk diffusion breakpoint determination for antimicrobial

Documents