challenges in the design and analysis of a randomized ... · pdf fileanalysis of a randomized,...
TRANSCRIPT
Challenges in the Design and
Analysis of a Randomized, Phased
Implementation (Stepped-Wedge)
Study in Brazil
Design of Experiments in Healthcare Conference
Isaac Newton Institute, Cambridge
Lawrence H. Moulton
Professor, Departments of International Health and Biostatistics
Johns Hopkins Bloomberg School of Public Health
www.LarryMoulton.com
Co-Authors: A. Pacheco3, V. Saraceni1, J. Golub4 , S. Cohn4, S. Cavalcante1,3,
B. Durovni1,2, R. Chaisson4.1Rio de Janeiro City Health Secretariat, Rio de Janeiro, Brazil, 2Federal University of Rio de Janeiro, Rio de Janeiro, Brazil, 3Fiocruz, Rio de Janeiro, Brazil, 4Johns Hopkins School of Medicine, Baltimore, United States © 2011 Lawrence H. Moulton
2
Outline
• Description: Brazil TB clinic-based study
• Analytic approach
• Statistical features of design
– Power: stepped wedge, unequal cluster size,
within-clinic correlation
– Randomization
• Special analytic features and results
3
Impact of Widespread Use of TB Preventive Therapy for Patients
with Access to Antiretroviral Therapy in Rio de Janeiro, Brazil:
A Phased Implementation Trial
4
Rio de Janeiro • Betina Durovni• Solange Cavalcante• Valeria Saraceni• Antonio Pacheco• Rita Ferreira• Giselle Israel• Vitoria Vellozo• Lilian Lauria
JHU• Richard Chaisson• Jonathan Golub• Larry Moulton• Silvia Cohn• Bonnie King• Anne Efron• Susan Dorman
THRio Study Team
Funding: Bill and Melinda Gates Foundation, NIAID, Fogarty International Center
5
Key features:
• Want to perform a strengthening of health services intervention, getting clinic personnel to do TB testing, and put TST positives on prophylactic isoniazid regimen
• 29 clinics, HIV+ clientele, most on HAART; clinics enter the intervention status two at a time every two months
• For samples size/power, need to account for:
--loss of efficiency due to stepped-wedge design
--loss of efficiency due to within-clinic correlation
--variable size of clinics
• Need to constrain the randomized order of entry, to avoid serious covariate imbalances over time
6
THRio Study Timeline
Stepped-Wedge Design
Intervention and
Follow-up Period
(for all clinics)
Sep 05 Jan 08
48 60
Dec 09
7
First—Need to Consider Analytic Approach
• Study will take place over 2.5 years, and there may be a strong
temporal trend in TB incidence
• Perfectly control for calendar time by comparing, ON EACH DAY, TB
incidence in clinics that are still in control status to incidence in clinics
that are in intervention status
• Assume Poisson process with time-varying intensity:
)exp( ititit zn
where itn is the person-days of exposure in the ith clinic on the tth day,
t represents the effect of the tth day
is the log rate ratio comparing those in the intervention status ( =1)iz
to those in control status ( =0)iz
8
Analytic Approach (continued)
• Condition on each day’s risk set; form partial
likelihood, comparing covariates of incident
cases to those of the other patients: eliminates
• Use gamma frailty model to account for within-
clinic correlation over time
t
9
iTd
iTY
idiYand
number of incident cases in the ith month in the intervention clinics and
number of persons at risk in those clinics, and
the cases and persons in both Treatment and Comparison clinics
in the ith month. Then the log-rank test statistic is given by:
28
1
28
1
[ ( / )]
11
i i
i i
T T i ii
T T i iii
i i i
d Y d YZ
Y Y Y dd
Y Y Y
Accounting for Wedging in the Power Calculations
Use log-rank weights to approximate partial likelihood
weighting of events:
10
EZ
SWZ
Accounting for Wedging, Continued
Inflation Factors for Stepped-Wedge Design
Calculate Z in two ways:
Assumes equal ratio 14 Treatment: 14 Comparison
clinics throughout the study
Assumes stepped-wedge design, 2 clinics entering
Intervention status every 2 months
Excel spreadsheet gives: / 1.2E SWZ Z
(virtually the same under null and alternative hypotheses)
So, for 80% power, instead of using 0.8416, use 0.8426 x 1.2 = 1.01
(nominal 93.8% power)
For Type I error of 5%, use 2.352 instead of 1.96 (nominal 1.9% size)
(=1.44 add’l
deff factor)
11
Estimation of k, the Coefficient of Variation
• See Hayes & Bennett 1999 Int J Epidemiology
• In each of 10 clinics, sampled 24 patient charts, obtained
historical information on TB incidence
• Calculation result based on random effects model: k = 0.10
• Seems too good to be true; if the true rate is 3.6/100 pyrs,
expect 95% of clinic rates to be between 2.9 and 4.3—so used
several larger values for k, and calculated power as a function of
them (next slide).
• Final adjustment— need an average person-years per
cluster for the sample size formula; clinic sizes vary widely, so
use harmonic mean of the anticipated person-years in the
clinics:
instead of the mean 595, used y=346.4 pyrs/cluster
12
Final Brazil Power Calculations
Fixed k (CV), error levels, and control rate, then solved for
intervention rate and hence effectiveness from the
formula:
CV=0.15 CV=0.20 CV=0.25
Control, Rate/100 pyrs 3.65 3.65 3.65
Intervention, Rate/100 pyrs 2.29 2.20 2.10
Effectiveness 37% 40% 42%
22222
2/ )/()](/)[()(1 TCTCTC kyzzN
where N is the number of clusters in each trial arm, y is the pyrs in each cluster
13
Constraining the Brazil Randomization
of Entry Order Into Intervention Status
There are
= 5.4 × 1026
distinct possible orderings of the clinics.
But we want to achieve close balance on clinic-months spent in control and intervention status with respect to a number of covariates: administrative region, existence of a DOTS program, mean CD4, number and proportion of patients on TB therapy, and education.
Use the approach of Raab (Stat Med 2001) and Moulton (Clin Trials 2004) to greatly reduce the above number of orderings to those that meet certain criteria with respect to the above variables.
29 27 25 23 21 19 17 15 13 11 9 7 5 3 1
2 2 2 2 2 2 2 2 2 2 2 2 2 2 1
14
Form of Proportional Constraints
In particular, for each jth covariate xij of the ith clinic, for
a given entry month ti, t = 1, 3, 5, … , 29, and a given
proportional covariate-specific tolerance cj, we
require that a given entry order satisfies:
i.e. the sum of the covariate values weighted by the
number of months in intervention status must be
within cj ×100 percent of that for control status.
Note: in this formulation, the last-entered clinic does not
contribute to trial results.
29 29
1 129 29
(1/(1 )) (28 ( 1)) / ( 1) (1 )
i i
j i ij i ij j
i it t
c t x t x c
15
Clinic-Level Covariates and
Constraints• Variable Meaning cj
• DOTS existence of a DOTS program 0.1
• AP1 dummy variables for AP unit 0.2
• AP2 0.2
• AP3 0.2
• AP4 0.4
• AP5 0.4
• MNCD4 mean CD4 count 0.1
• NPTS number of current patients 0.1
• PCTINF proportion being treated for TB 0.1
• EDLT4 proportion < 4 yrs’ education 0.1
16
City of Rio de Janeiro
Health units - CREATE:
6 units
AP 1.0
AP 2.1
AP 2.2
AP 4.0
AP 5.1AP 5.2AP 5.3
AP 3.3
AP 3.1AP 3.2
< 100 patients
1 units
100 – 500 patients
500 - 1000 patients
> 1000 patients
1 unit
3 units
1 unit
8
4
3, 7, 9
112
10
5
4
2 34
6, 10, 134, 5, 8
3
5 59
6
13
17
4
7
In purple—TB incidence rates
17
Sequence Generation
• We want to select, at random, one sequence from among
all those that satisfy these criteria. If there were a
reasonable number of total sequences (not 10**26) then
we would check each one to see which meet the criteria,
then randomly choose one of the eligible sequences.
• Equivalently, we could randomly generate sequences,
testing each one until we get an acceptable one.
• What we actually did was generate sequences until we
had 1000 that were acceptable; then carried out the final
selection of one of these in a quasi-public randomization
ceremony.
• Method for selection of the final one out of the 1000
eligible:
Specified that we would take the last three digits of the
26 February 2005 Loteria Federal to select it!
18
Design Validity
• Generally: A design is valid if each pair of randomization units has the same probability of being allocated the same treatment (RA Bailey 1983 Biometrika)
• In our one-way crossover design—we want each pair of units to have (about) the same probability of entering intervention status at the same time
• Simulations show it takes a high degree of departure from uniformity of pairing probabilities, coupled with high correlation of affected randomization units, to significantly impact Type I error (Moulton, 2004 Clinical Trials)
19
Tradeoff
Invoke minimization (Pocock & Simon,1975 Bics) in a simultaneous fashion, randomly selecting one of the sequence allocations from among those that have good balance.
* Want to constrain enough so that ensure marginal balance on relevant covariates—like minimization technique for sequential trials except do it all up front
• Don’t want to constrain so much that validity (independence) is lost, setting one up for criticism
20
Validity Check
• Randomly generated sequences until had 5,000 acceptable designs. This took 8.96×10**7 random attempts; that means we estimate that only about 5000/89,600,000, or 0.006% of the total possible designs are acceptable.
• No duplicates among the 5000, although there had been 471 duplicates in a previous run that had the precision for AP4 and AP5 set at 0.3 instead of 0.4
21
Validity Check (continued)
The following matrix gives the number of times that any given pair of clinics entered the trial at the same time point. For example, clinics 3 and 7 entered in the same time point 132 times out of 5,000. All 5,000 x 14 = 70,000 entered pairs are ―folded‖ into the upper diagonal; there are 406 upper diagonal entries, and we would expect about 70,000/406 = 172 in each entry if there were no constraints, with a 95% confidence interval of ± 26.
Although there is an almost three-fold ratio of the highest to the lowest entries, all pairs of clinics had multiple opportunities to enter together, and no pairs never entered together.
22
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 1 0.000 104. 117. 132. 124. 114. 131. 136. 121. 148. 138. 149. 151. 172. 163. 163. 197. 172. 168. 210. 22 0.000 0.000 117. 99.0 127. 110. 128. 137. 120. 139. 153. 150. 178. 151. 177. 166. 198. 181. 203. 183. 23 0.000 0.000 0.000 127. 114. 116. 132. 126. 155. 140. 168. 135. 168. 150. 167. 179. 184. 193. 180. 174. 196. 197. 201. 236. 228. 205. 232. 234 0.000 0.000 0.000 0.000 138. 122. 137. 110. 134. 146. 149. 170. 154. 169. 173. 166. 188. 200. 203. 193. 25 0.000 0.000 0.000 0.000 0.000 140. 120. 140. 147. 146. 165. 159. 191. 187. 183. 177. 169. 178. 188. 203. 16 0.000 0.000 0.000 0.000 0.000 0.000 139. 128. 162. 155. 168. 156. 150. 170. 185. 184. 193. 180. 188. 171. 27 0.000 0.000 0.000 0.000 0.000 0.000 0.000 150. 151. 168. 143. 181. 176. 186. 157. 171. 191. 175. 203. 181. 18 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 168. 156. 171. 141. 172. 182. 179. 190. 162. 220. 191. 200. 19 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 161. 165. 193. 172. 166. 159. 153. 192. 175. 183. 179. 110 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 170. 157. 201. 139. 190. 166. 170. 170. 181. 170. 111 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 168. 177. 170. 155. 154. 165. 158. 160. 191. 112 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 142. 190. 178. 169. 171. 157. 175. 163. 113 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 162. 160. 184. 166. 159. 179. 180. 114 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 176. 165. 168. 166. 172. 175. 115 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 159. 164. 183. 183. 186. 116 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 144. 168. 193. 209. 117 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 178. 164. 218. 118 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 176. 179. 119 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 178. 120 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 121 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.22 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.23 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.24 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.25 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.26 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.27 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.28 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.29 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.
“Validity Matrix”
23
Clinic-Level Covariates and Constraints
• Variable Meaning cj Actual
• DOTS existence of a DOTS program 0.1 0.03
• AP1 dummy variables for AP unit 0.2 0.10
• AP2 0.2 0.04
• AP3 0.2 0.10
• AP4 0.4 0.09
• AP5 0.4 0.21
• MNCD4 mean CD4 count 0.1 0.08
• NPTS number of current patients 0.1 0.07
• PCTINF proportion being treated for TB 0.1 0.07
• EDLT4 proportion < 4 yrs’ education 0.1 0.07
24
Final Comments on Brazil Design
• ―…we make no mockery of honest ad hockery‖ --I.J. Good
• Had to compensate for many factors in the sample size calculation: unequal group sizes, correlation across time, time-varying ratios of trial arm sizes (due to phased intervention)
• Used a rather simple, although greatly restraining, set of constraints for the randomization; other weighting systems might be of interest
• Phased implementation design both suitable and logistically feasible
Primary Analysis
• Based on when patients had the
opportunity for the intervention—when
they showed up in a clinic that is in the
intervention phase
• Mimics long-term effectiveness of the
intervention.
Statistical Analysis
• Cox model with clinic-level random effects
• Main covariate of interest:
– For each patient, binary time-varying
covariate tracks switch from control to
intervention phase for that patient
• Simulations showed that this gives same
result as time-varying covariate that is the
proportion of individuals in a clinic who
have shown up during intervention phase
– (as per basic idea in Hussey & Hughes, Contemp Clin Trials 2007)
Slight Problems
• Don’t have dates for all clinic visits—but
will use any information indicating there
was a meaningful clinic contact (e.g. CD4
result, HAART begun,…)
• Will not count a TB case unless the patient
has been in the clinic at least 60 days, so
that prevalent TB is not included—same
for deaths.
Which TB Cases Count for
What?
• Control Phase
– If TB is diagnosed before a patient’s first clinic
visit when the clinic entered intervention
phase
• Intervention Phase
– If TB is diagnosed after a patient’s first clinic
visit during intervention phase
30
Cox Model Looks at Cross-Sectional
Data Slices
1/2
3/4
5/6
7/8
9/1
0
2
9
Clin
ic e
ntr
y to inte
rvention p
eriod
Control
phase
Intervention
phase
Follow-up
period
1 3 5 7 9 29 36 42
Month
Note: still making vertical comparisons during final follow-up period,
as patients are still showing up for their first time in intervention phase
(Perfectly accounts for secular trends)
Analysis at Individual Level
• Clinics are handled as Intent-to-Treat
• Individual patients do not count in the
―treat‖ part until they show up at a clinic
which has already received the
intervention.
• Patients tracked using survival analysis
setup; calendar timeline is used.
• Dynamic cohort—patients entering all the
time, transferring, dying…
02
46
8
Inte
rven
tion
TB
Incid
en
ce /1
00
pyrs
0 2 4 6 8Control TB Incidence /100 pyrs
Circles below line indicate a clinics with lower rates during intervention phase.
(Area is proportional to the inverse variance of the rate difference)
36
Pure ―Vertical‖ TB Rates Analysis
37
0
1
0
1
0
1
0
1
0
101
01
0
1
01
0
1
0
1
0
1
0
1
0
1
1
0
.02
.04
.06
rate
s
0 5 10 15entryinterval
TB incidence rates per person-year by each two-month interval.
1 is for intervention, 0 is for control phase units. The last interval is
actually from when all units are in intervention until Aug 2009.
Clinic Rate Differences by Order of Entry into Intervention
--another check on secular trend effects
38
0
1x10-5
2x10-5
3x10-5
4x10-5
5x10-5
0 200 400 600
Follow-up Time
Hazard
Ratio
Courtesy V.Saraceni & K.Lin 41
THRio Results
TB cases, total contribution time, incidence per 100pyrs
Control Phase Intervention Phase
Cases 221 254
Person years 16,834 23,126
Rate/100pyrs 1.31 1.10
43
TB/Death cases, total contribution time, incidence per 100pyrs
Control Phase Intervention Phase
Cases 617 696
Person years 16,834 23,126
Rate/100pyrs 3.67 3.01
Conclusions
44
Stepped wedge approach can be fraught
with complications: -- Loss of power
-- Balancing randomization
-- Handling lag in implementation of
intervention: neither vertical nor
horizontal, but diagonal, analysis may be needed.
These all require careful handling and cross-checks.
Other potential problems: -- Secular trends may still linger
-- Aging of cohort confounded with
one-way crossover
THRio had political and logistical reasons for this
design; but better to use a parallel design if
possible!