Kathleen Carroll & Brian KilukDivision of Substance Abuse

Yale University School of Medicine

Supported by NIDA Supplement to R01 DA15969

and P50 DA09241, U10 DA015831, R01 DA019078, & R01 DA 10679

Why do we need a sound and valid indicator? Facilitation of comparisons across

projects, meta-analyses Set and monitor performance

standards Benchmarking Clearly convey magnitude of treatment effects to

stakeholders Facilitate comparisons across common standard Lack of incentive to improve performance and

outcome (retention not appropriate standard)

Overview

Desirable characteristics of indicators

Strengths and weaknesses of common approaches

Overview of our project

“Traditional” indicators of clinical significance almost always translate to complete abstinence Return to normative levels Reliable change indices Return to healthy functioning? (e.g.,equivalent

of ‘no heavy drinking days’ for stimulant users)

What are we looking for in an indicator?

Easy to calculate, interpret Psychometrically sound, reliable, replicable Low susceptibility to missing data Verifiable (biologic indicator, other) Independence from baseline measures Sensitive to treatment effects Low(er) cost Predicts long-term cocaine outcomes Related to indicators of good long term functioning Acceptable to field Easily interpreted by clinicians, policy makers, payors

What is ‘success’ in treating stimulant users? Durable periods of abstinence

Employment, productivity Lack of criminal activity Reduced use of expensive, avoidable health care

resources

11% at end of treatment, 21% at end of 1 year follow up

Why not complete abstinence? Insensitive to change Difficult standard for most individuals

(14% of our sample of 434) Chronically relapsing disorder Change is dynamic Starting and remaining abstinent may imply

questionable need for treatment Our data: Weak relationship with cocaine

use and functioning outcomes at one year

Retention

ProsEasy to calculateAvailable for all participantsIndicator of treatment acceptabilityIndicator of differential attrition/data availability across conditions

ConsMay be more meaningful in some contexts than othersParticipants leave treatment for different reasonsIs retention with continued use meaningful?Is compliance with ineffective treatment meaningful?Not related to long-term outcome in our sample

Percent negative urines ProsWidely used and acceptedLess susceptible to demand characteristics, misrepresentationQuantifiable, ability to detect new episodesVery accurate, if appropriate schedule of collection and minimal missing dataTiming is critical (overlap, missing data)

ConsRecent use only (3-5)High cost for frequent or quantitativeSensitive to missing data, esp. with differential attritionDepends on assumptions (missing, denominator)Stimulants or all drugs?Can’t back-fillProblems with assuming missing=positive*

Calculating percent urine samplesExample: 1 negative urine, 2 sessions, then dropout of 12 week trial.

Based on submitted: 100%

Based on possible: 50%

Based on expected/ 1x 8%

Based on expected/ 3x 3%

Percent cocaine positive 0%

Longest consecutive x-free urine specimens ProsStrong evidence of meaningful abstinenceLess susceptible to demand characteristics, misrepresentationQuantifiable, ability to detect new episodesVery accurate, if appropriate schedule of collection and minimal missing dataTiming is critical (overlap, missing data)

ConsHigh cost for frequent or quantitativeVery sensitive to missing data, esp. with differential attritionDepends on assumptions (missing, denominator)Stimulants or all drugs?Can’t back-fill

Percent days abstinent, self reportProsWidely usedPotentially available for all participants and all days if TLFB used with high data completion; highly flexibleTrue intention to treat possibleCan be reliable if methods to enhance reliability used (at a cost)Our discrepancy rate=8-12%

ConsWith high/differential dropout, what’s the denominator? Days in treatment versus days expected?Not easy to correct with urine data if discrepancies high

Maximum days of abstinence,overall or in final x weeksProsLinked linked to longer-term cocaine usePotentially verifiable if urines collected at appropriate intervalsProvides ‘grace period’Easily dichotomized (eg 3 plus weeks)

ConsHigh complexity with missing data, especially dropoutsHigh complexity if discrepant urine data Participants last 2 weeks or last 2 weeks of trial?End of treatment or sometime within treatment?

Reduction in use: Frequency and or quantityProsAlternative to abstinence; more achievable target?Highly compatible with random regression modelsSensitive to treatments that may take time for effects to emergeProvides ‘grace period’Easily dichotomized

ConsComplexity obtaining accurate estimates of frequency/quantity of use prior to baselineWhen is reduction measured (last weeks? Entire course?Costs of repeated quantitative urines, sensitivity to missing data

Issues in defining ‘reduction’ Patterns vary widely (binge versus low

use) Reliable estimation of quantity complex

(illicit, no standard units, adulterants common, potency varies, hard to standardize ‘hits’ ‘joints’ ‘dime bags’)

Difficulty of estimating dollar value (commerce, shared use, sex for drugs)

Which indicator of treatment response?Loss of power with dichotomous, but also easily interpretable, calculable for all, relevant to clinical significanceCandidates

*Complete abstinence*3 or more weeks of abstinence*End of treatment abstinence*Reduction of x percent“Good functioning”

 

IndicatorEase of

computationVerifiability Vulnerability to missing

dataRelative cost Operationalization

for these analyses

1Days retained in treatment protocol C

Easy Yes- Low Low Days from randomization to endpt

2

Percentage of urine specimens testing positive C

Easy for complete data Yes, by definition Assumes independence of urine specimens

(denominator), assumes numerator is unbiased by

collection schedule or missing data.

High Number of cocaine-negative urine

specimens collected / all specimens collected

3

Maximum consecutive days abstinent

C Easy for complete data Yes, provided appropriate schedule of

data/urine collection

Likely to result in casewise missingness or reduced

sample size

Moderate, due to biological

verification and derivation from

TLFB

Longest continuous cluster of self-reported

abstinence within treatment

4

Percent days of abstinence from cocaine C

Depends on treatment duration, level of missing data, and

intermittent missingness

Yes, provided appropriate schedule of

data/urine collection

Likely to result in casewise missingness or reduced

sample size

Moderate, due to biological

verification and derivation from

TLFB

Number of self-reported days of abstinence from

cocaine / days in treatment (retention)

5

Maximum days of continuous abstinence during last two weeks of treatment C

Complex for intermittent and monotone,

dropouts

Yes, provided appropriate schedule of

data/urine collection

Low Moderate, due to biological

verification and derivation from

TLFB

For those retained 14+ days, longest cluster of

abstinence in final 2 weeks; otherwise 0

6

Completely abstinent last two weeks of treatment D

Easy Yes, provided appropriate schedule of

data/urine collection

Low Moderate, due to biological

verification and derivation from

TLFB

For those retained 14+ days, 0 days of use in last 14 days, otherwise

0

73 or more weeks of continuous abstinence D

Easy Yes, provided appropriate schedule of

data/urine collection

Low Moderate, due to biological

verification and derivation from

TLFB

“Yes” if participant retained 21+ days, max

days abstinent > 20. Otherwise No

82 or more weeks of continuous abstinence D

Easy Yes, provided appropriate schedule of

data/urine collection

Low Moderate, due to biological

verification and derivation from

TLFB

“Yes” if participant retained 14+ days, max

days abstinent > 13. Otherwise No

  Note. C=continuous, D=Dichotomous, TLFB=Timeline Followback method 

 

IndicatorEase of computation Verifiability Vulnerability to

missing dataRelative cost Operationalization for

these analyses

91 or more weeks of continuous abstinence D

Easy Yes, provided appropriate schedule of

data/urine collection

Low Moderate, due to biological verification and derivation from

TLFB

“Yes” if participant retained 7+ days, max days

abstinent > 6. Otherwise No

10

Completely abstinent from cocaine during treatment

D Easy Same Low Moderate, due to biological verification and derivation from

TLFB

0 days of use and 0 positive urines

11

Completed treatment and abstinent in last week D

Easy Yes Low Low Completion of treatment, 0 days of use in final week

12

Percent reduction in frequency of use (28 days prior/days last 4 weeks) C

Complex, baseline definition can be

arbitrary

No, relies on accurate baseline/pretreatment

assessment

Moderate Low Percent days of use in final 28 days of treatment/

percent days of use in 28 days prior to baseline

1350% reduction in frequency of use D

Complex, baseline definition can be

arbitrary

Relies on access to accurate

baseline/pretreatment level of use

Moderate Low % reduction is 50% or higher

1475% reduction in frequency of use D

Complex, baseline definition can be

arbitrary

Same Moderate Low % reduction is 75% or higher

15

Report no drug use, legal, employment, or psychological problems last 28 days of treatment D

Easy Partial Low Low Completes treatment, 0 days of problems in drug,

legal, employment and psych ASI in past 28 days

  Note. C=continuous, D=Dichotomous, TLFB=Timeline Followback method

 

IndicatorEase of

computationBiological

verificationVulnerability to missing

data

Sensitivity to

treatment effects

Relationship with post

tx outcomes

Independent from baseline

indicators

Relationship to measures of

general function

21-30 days of abstinence

X Relies on appropriate

schedule

Low

Completed treatment and abstinent in last

2 weeks

X Same Low

50 % reduction Complex, baseline

definition can be arbitrary

Relies on having accurate baseline/pretrea

tment assessment of

use

Moderate

% days abstinent

Depends on treatment duration,

complex for dropouts, and intermittent missingness

X, provided appropriate schedule of data/urine collection

Moderate

Max days consecutive abstinence

Complex for intermittent and

monotone missingness,

dropouts

X, provided appropriate schedule of data/urine collection

Likely to result in casewise

missingness or reduced sample

size

Percent neg ative urine specimens

Easy except when missing

data

yes

So far…

Carroll, K.M., Kiluk, B.D., et al. (2014). Towards empirical identification of a reliable and clinically meaningful indicator of treatment outcome for illicit drug use. Drug and Alcohol Dependence, 137, 3-19.

Kiluk, B.D., et al. (2014). What happens in treatment doesn’t stay in treatment: Cocaine abstinence during treatment is associated with fewer problems at follow-up. J Consulting and Clinical Psychology, 82:619-27. 

DeVito, E.E., et al. (2014). Gender differences in clinical outcomes for cocaine dependence: Randomized clinical trials of behavioral therapy and disulfiram. Drug and Alcohol Dependence, 145: 156-167.

Decker, S.E., et al. (2014). Assessment concordance and predictive validity of self-report and biological assay of cocaine use in treatment trials. The American Journal on the Addictions, 23, 466-74. 

Kiluk, B.D., et al. (in press). Prompted to treatment by the criminal justice system: Relationships with treatment retention and outcome among cocaine users.