secondary analyses in clinical trials
DESCRIPTION
SECONDARY ANALYSES IN CLINICAL TRIALS. Presented by: George Bigelow, PhD Daniel J. Feaster, PhD Abigail G. Matthews , PhD December 6, 2013. Objectives. Review the statistical issues with analyzing and interpreting secondary analyses, and demonstrate the multiple testing burden - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/1.jpg)
2013 CTN Web Seminar Series
Produced by: NIDA CTN CCC Training Office"This training has been funded in whole or in part with Federal funds from the National Institute on Drug Abuse,
National Institutes of Health, Department of Health and Human Services, under Contract No.HHSN271201000024C."
SECONDARY ANALYSES IN CLINICAL TRIALS
Presented by:George Bigelow, PhDDaniel J. Feaster, PhD
Abigail G. Matthews, PhD
December 6, 2013
![Page 2: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/2.jpg)
2
Objectives• Review the statistical issues with analyzing
and interpreting secondary analyses, and demonstrate the multiple testing burden
• Explain the importance of secondary outcome and analysis identification during protocol development
• Discuss reporting and interpretation of secondary analyses, including the perspective of the CTN Publications Committee
![Page 3: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/3.jpg)
3
Outline• Introduction and motivation• Statistician’s perspective• Multiplicity• Implementing secondary analyses• Summary• Discussion
![Page 4: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/4.jpg)
4
PUBLICATIONS COMMITTEE PERSPECTIVE ON SECONDARY ANALYSES: OPPORTUNITIES & CAUTIONS
![Page 5: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/5.jpg)
5
Opportunities• CTN encourages multiple publications• We want to learn as much as possible• Large-N, diverse, multi-site studies• Broad study teams with diverse interests• Extensive investment in assessments• Repeated assessments over time• Assessment commonalities across studies
![Page 6: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/6.jpg)
6
Cautions• CTN studies typically yield multiple
publications• Question: Are we over-analyzing the data?• Discussions in Publications and Executive
Committees• Consequence: This webinar
![Page 7: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/7.jpg)
7
Cautions (cont.)• Multiple testing incurs risk of false
conclusions• Proper planning can reduce this risk• Acknowledgement of limitations is essential
![Page 8: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/8.jpg)
8
Example of Multiple Publications• CTN006/007: MIEDAR – Abstinence-
Contingent Incentives• Report of primary outcome• Do contingent incentives reduce stimulant
use?
![Page 9: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/9.jpg)
9
Example of Multiple PublicationsCTN006/007:
MIEDAR – Abstinence-Contingent IncentivesReports of secondary outcomes:
Do contingent incentives affect……HIV risk behavior?
…gambling?…cost or cost effectiveness?…methamphetamine use?
…staff attitudes?
![Page 10: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/10.jpg)
10
Example of Multiple PublicationsCTN006/007: MIEDAR – Abstinence-Contingent
Incentives
Reports of moderator variable associations: Are incentive effects related to…
…gender, race, ethnicity?…treatment history?
…criminal justice involvement?…urinalysis result at intake?
…gambling history?
![Page 11: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/11.jpg)
11
A Caution About Demographic Subgroup DifferencesBe cautious of thinking of subgroup differences
as inherent characteristics of those groups or of individuals within those groups
Demographic subgroup differences are very likely the result of some correlated
confounding variable; they likely reflect differences in life experiences and
opportunities, and the contexts in which drugs are encountered
![Page 12: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/12.jpg)
12
Example of Multiple PublicationsCTN006/007: MIEDAR – Abstinence-
Contingent Incentives
Reports of associations unrelated to the study intervention:
What symptoms are related to dependence on various drugs?
![Page 13: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/13.jpg)
13
Types of Analyses• Intervention Effects on Primary Outcome
• Intervention Effects on Secondary Outcomes
• Analysis of Moderators or Mediation
• Associations
![Page 14: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/14.jpg)
14
Common Errors• Mistaking correlations for causes
• Mis-describing the study methods
• Overlooking explanatory confounding variables
• Failing to acknowledge limitations
![Page 15: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/15.jpg)
15
Correlation is Not CausationAvoid language that implies causality when reporting associations
Examples
“effect on” “effect of”
“impact on” “impact of”
“consequently” etc.
![Page 16: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/16.jpg)
16
Describe Methods Accurately• Understand and describe original study
accurately
• Explain origins and methods of secondary analysis
• Idea should precede looking through the data
• Describe types and numbers of analyses performed
![Page 17: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/17.jpg)
17
Consider Confounding Factors• One report examined relationship between
study pay and proportion of Ss present at the final assessment
• Proposed implausible conclusion that greater pay led to less retention
• Failed to note that pay amount was related to study duration and difficulty
![Page 18: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/18.jpg)
18
Proceed with CautionMany of the factors that must be considered in conducting and reporting secondary analyses are the same as those important for careful and thoughtful reporting of primary analyses
However, secondary analyses can also involve some special statistical considerations, as will be discussed by the following speakers
![Page 19: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/19.jpg)
19
STATISTICIAN’S PERSPECTIVE AND THE ISSUE OF MULTIPLICITY
![Page 20: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/20.jpg)
20
What do we mean by primary and secondary outcomes and analyses?
• The primary outcome is the main outcome variable for the study– Research hypothesis based on this measure– Used to power study and determine statistical
significance of any treatment effect – Analytic method must be specified a priori in
the Statistical Analysis Plan (SAP) at a minimum
![Page 21: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/21.jpg)
21
What do we mean by primary and secondary outcomes and analyses?
• Secondary analyses are any other analyses, e.g.:– Sensitivity analysis of primary outcome
measure with respect to missing data– Subgroup analyses by age, race, gender,
ethnicity, disease severity, etc.• Secondary outcomes are any other
outcome measures
![Page 22: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/22.jpg)
22
Why secondary analyses?• Publish or perish!!!• Possible that primary outcome measure
data ends up being unreliable– e.g., using TLFB but high rate of discordance
between self-report and UDS• Analytic issues with pre-specified primary
analysis– e.g., proposed distribution of the primary
outcome variable does not hold and alternative methods should be used
![Page 23: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/23.jpg)
23
Why secondary analyses? (cont.)• Possibly poor power of primary analysis if
assumptions used in sample size are not appropriate
• Sensitivity analyses of primary outcome with respect to missing data – key for addiction research
• Subgroup analyses– Race, ethnicity, gender required by NIH– Baseline severity of disease (Nunes et. al., 2011)
SPECIFY A PRIORI IN PROTOCOL OR SAP AS MUCH AS POSSIBLE!
![Page 24: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/24.jpg)
24
Why secondary outcomes?• Publish or perish!!!• In addiction research we always focus on
abstinence but is that enough?– Improved overall quality of life– Engaging in less risky sexual behaviors– Less illegal activity such as theft or
prostitution– CTN TEAM Task Force recommends at least
one secondary outcome measure be related to “functioning, satisfaction, or quality of life”
![Page 25: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/25.jpg)
25
Why secondary outcomes? (cont.)• Again, what if there are unanticipated
issues with the primary outcome? • Cannot “hang your hat” on only one
outcome measureSPECIFY A PRIORI IN PROTOCOL OR SAP AS MUCH
AS POSSIBLE!
![Page 26: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/26.jpg)
Example of Utility of Secondary Analyses• CTN: Women with Trauma and Addictions• Primary outcome results (Hien et. al., 2009):
– Trauma symptom severity: NS– Abstinence: NS
• Secondary analyses:– Women with baseline eating disorders had significantly
less improvement in PTSD severity and abstinence– SS significantly reduces unprotected sex in high risk
women over time– Racial/ethnic matching with therapist associated with SS
effectiveness– Examples of other positive findings: retention, sleep
disorders, intimate partner violence
![Page 27: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/27.jpg)
Denise Hien CPDD 2013 Presentation:
![Page 28: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/28.jpg)
28
Words of Warning• Too many post hoc analyses opens one to
accusations of data dredging
• Secondary analyses/outcomes cannot be used to evaluate the trial as a whole (only primary outcome)
• If there are a substantial number of pre-specified secondary outcomes and analyses, consider adjusting for multiple comparisons
• Appropriate interpretation of results is key– Hypothesis generating
![Page 29: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/29.jpg)
29
Cautionary Tale
Scott Harkonen, MD
• Convicted of wire fraud: “willfully overstating in a press release the evidence for benefit of a drug his company made”
• Primary outcome p-value=0.08• Asked his statisticians to identify sub-group with significance • Patients with mild to moderate disease severity: p=0.004• Press release acknowledged negative finding from primary
outcome analysis but maintained drug associated with increased survival
• Post: “…everyone agrees there weren’t any factual errors in the four-page document. The numbers were right; it’s the interpretation of them that was deemed criminal.”
• During appeal court said: “Statements are fraudulent if ‘misleading or deceptive’ and need not be literally ‘false’.”
![Page 30: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/30.jpg)
30
Why such controversy?Multiplicity• Type I error is preserved (usually 5%) for primary
outcome(s)
• If performing multiple secondary analyses, then overall Type I error will be higher
• If enough analyses are performed, there will be at least one spurious association
• Adjustment not necessary for secondary analyses/outcomes, but interpretation must be cautious and presentation of results forthright and transparent
![Page 31: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/31.jpg)
31
Illustration of Multiplicity• Generate 10 outcome variables
independently from normal with mean=0 and variance=1 for 300 participants
• Calculate Spearman correlation coefficient for each pair-wise combination
• Test correlation coefficient ≠ 0
• Type I error estimated as the number of tests that are statistically significant divided by number of tests (45)
![Page 32: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/32.jpg)
32
Illustration of Multiplicity (cont’d)V2 V3 V4 V5 V6 V7 V8 V9 V10
V1 R = 0.03p = 0.661
R = -0.03p = 0.541
R = 0.04p = 0.448
R = 0.16p = 0.006
R = 0.07p = 0.231
R = -0.04p = 0.481
R = <0.01p = 0.960
R = 0.01p = 0.875
R = -0.08p = 0.186
V2 R = -0.05p = 0.436
R = 0.03p = 0.548
R = -0.12p = 0.033
R = 0.06p = 0.310
R = 0.02p = 0.793
R = 0.10p = 0.083
R = 0.01p = 0.877
R = 0.07p = 0.222
V3 R = 0.10p = 0.082
R = 0.05p = 0.427
R = 0.07p = 0.197
R = -0.03p = 0.605
R = -0.06p = 0.322
R = -0.08p = 0.149
R = 0.07p = 0.197
V4 R = -0.04p = 0.494
R = 0.08p = 0.149
R = -0.14p = 0.013
R = 0.15p = 0.012
R = 0.03p = 0.598
R = 0.05p = 0.396
V5 R = 0.06p = 0.282
R = -0.03p = 0.550
R = -0.04p = 0.501
R = 0.02p = 0.772
R = -0.16p = 0.005
V6 R = 0.05p = 0.433
R = 0.05p = 0.397
R = -0.14p = 0.013
R = -0.06p = 0.273
V7 R = -0.11p = 0.055
R = -0.07p = 0.227
R = -0.09p = 0.116
V8 R = 0.03p = 0.614
R = -0.03p = 0.623
V9 R = 0.07p = 0.232
Type I Error Rate = 6/45 = 13.3%
![Page 33: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/33.jpg)
33
Implications• Avoid post hoc analyses
• Pre-specify as much as possible (protocol or SAP) →
→ Avoid data dredging criticism
→ Can even adjust Type I error rate for number of secondary analyses performed (rare)
• Interpret secondary results keeping in mind the inflated Type I error rate
![Page 34: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/34.jpg)
34
Responsible Analysis and Reporting• Focus should always be on primary outcome
• Of secondary analyses, focus should be on those that were pre-specified
• Requires careful planning with statement of hypotheses in protocol/SAP (SAP should be finalized before data lock)
• Report in a manuscript the number of pre-specified analyses performed and the number reported
![Page 35: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/35.jpg)
35
Responsible Analysis and Reporting (cont’d)• Present estimates of treatment differences
and CIs:“plausible range of treatment differences consistent with trial results”
• Interpretation needs to be viewed as exploratory rather than confirmatory
• Frame results in context of supporting or contradictory data from other studies
![Page 36: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/36.jpg)
36
Responsible Analysis and Reporting (cont’d)• For post hoc analyses:
– Acknowledge that analyses were not specified a priori (data driven)
– Describe why analyses are important and the relevance of the research question
– Report number of post hoc analyses performed and the number reported
– Significance should be viewed as descriptive and not used for inference or decision making
– Can be used to justify future research
![Page 37: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/37.jpg)
37
ExamplesIf primary outcome not statistically significant but some pre-specified secondary analyses were:
While the primary outcome did not demonstrate statistically significant evidence of a treatment effect, some secondary analyses suggested that the treatment may be effective. Therefore, future research is warranted.
If primary outcome is statistically significant but no secondary analyses are:
The primary outcome was statistically significant indicating that treatment is effective in this study population. Despite the fact that numerous secondary analyses did not yield statistical significance, there is sufficient evidence to justify future research of this intervention.
![Page 38: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/38.jpg)
38
Questions…
![Page 39: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/39.jpg)
39
IMPLEMENTATION OF SECONDARY ANALYSES
The Design Stage
![Page 40: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/40.jpg)
Multiple Types of Secondary Analyses• Secondary hypotheses—Utilize the design• Mediation studies—Use data post-
randomization• Association Studies—Normally don’t use
the design– Example: Predictors of HIV Testing
(CTN0032)– Since do not use the study design—
observational!40
![Page 41: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/41.jpg)
Multiple Types of Secondary Analyses (cont.)• Subgroup or Moderator Analyses
– Risk reduction counseling impact on HIV testing by modality of substance use treatment (CTN0032)
– Differential Treatment Effects by Race/Ethnicity and/or Gender
– We do not randomize to subgroups—observational!
41
![Page 42: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/42.jpg)
Why consideration at design stage?• Appropriate measures
• Sample size considerations
• For secondary analyses that do NOT use design:– Causal interpretation is difficult– Statistical models can help
• Subject to assumptions—no unmeasured confounders• Implies need to think about and measure confounders for
any secondary analysis that is not just a test of difference by randomized treatment group
42
![Page 43: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/43.jpg)
Secondary Outcomes• Simplest type of “secondary” analysis
• Other outcomes on which we feel the intervention will have an impact
• Analysis strategy frequently very similar to primary outcome analysis
• Need to consider multiple testing issue!– If enumerate and measure 20 secondary outcomes,
the Type 1 error is .64 (if we use α = .05 for each test)– Each secondary outcome being in a separate paper
does NOT change this fact43
![Page 44: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/44.jpg)
Mediation Analyses• Another analysis frequently included in
protocols• Because pieces of the model are determined
after randomization, there is difficulty making a strong causal interpretation
44
Treatment Assignment
Attendance in Treatment
Count of Drug Use Days
![Page 45: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/45.jpg)
But we do not randomize attendance, so even if observed for everyone:
45
Treatment Assignment
Attendance in Treatment
Count of Drug Use Days
CONFOUNDERS
We cannot rule out confounders of both Attendance and Drug Useand, therefore, cannot make strong causal statements about the gold pathway without strong assumptions.
![Page 46: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/46.jpg)
Possible AssumptionsNo Unmeasured Confounders
• If we measure all potential confounds, then can make causal statements SUBJECT to the assumption that we have measured ALL the confounds
• May want to measure potentially confounding factors
• Example: Propensity Score Analysis
Instrumental Variables
• If we can find exogenous factors, Z, that are correlated with Attendance, X
• Also, Z does not directly affect Count of Drug Use Days
• Can use Z as instruments to identify the causal impact of attendance on Count of Drug Days
• Use the instrumental variable in place of the endogenous variable
46
![Page 47: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/47.jpg)
Association Studies• Like mediator models, association studies are
looking at the impact of variables which the experimenter has not controlled on some chosen outcome→Do not make causal claims (unless include confounder analysis or Instrumental Variable(s))!
• Frequently these studies will look at numerous predictors– Type I error– Must be very clear and honest about the way the
analyses were done47
![Page 48: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/48.jpg)
Potential Solution for Type I Error• Machine Learning approaches
– Used in data mining– Allows an exhaustive search for the best
predictive model of the outcome– Like testing all covariates, sometimes can over-fit
• Cross-validation– Simplest approach is split sample, explore on
one sample, then replicate on a second sample (the second sample is a “test” of results on the first sample)
48
![Page 49: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/49.jpg)
Subgroup or Moderator Analyses: Should Determine Which Subgroups Are of Interest at Design Stage• Many of us have interests in:
– Racial and ethnic groups– Gender
• But other subgroups may be of interest– Drug of choice– Severity of individual’s problem – Age– Socioeconomic status – Site Differences (in levels of outcome and/or treatment
effects)– PTSD/No PTSD
• Important to define groups a priori (Necessary to consider at the design phase) 49
![Page 50: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/50.jpg)
Many (if not most) subgroups are not randomized.
• This means these are observational models and cannot make causal statements– Should assess for confounds– Should be careful not to over-interpret– Even if assess for confounds, cannot rule out
unobserved confounds– Race/Ethnic differences are examples where
differences are largely NOT causal (race/ethnicity is correlated with true casual agent)
50
![Page 51: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/51.jpg)
Example
• If results show there is an interaction & appears that treatment works best in high severity
• Cannot be sure that high severity “caused” treatment success (or that treatment will work in high severity) because have not randomized to high severity 51
Count of Drug Use Days
Randomized Group
Baseline Drug Use Severity
Randomized Group X Baseline Drug Use Severity
Genetic factor
![Page 52: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/52.jpg)
Suggestions• Pre-specify at the design stage the subgroups of
interest
• Plan to assess known confounds related to the subgroups
• Minimize the number of subgroups examined (Type I error issue)
• Use tests of interaction within all participants, rather than testing treatment effects within each subgroup!!!
52
Wang et. al., Statistics in Medicine – Reporting of Subgroup Analyses in Clinical Trials. NEJM, 2007:357;21.Lagakos, S. The Challenge of Subgroup Analysis—Reporting without Distorting. NEJM, 2006:354:16.
![Page 53: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/53.jpg)
Defining Subgroups• Easiest to work with subgroups that are
inherently categorical (race/ethnicity, gender, primary drug of use, site)
• Subgroup membership is ambiguous (and potentially manipulated) if defined on continuous measures (age or income, etc.)—better to include a continuous interaction
• Focusing on categorical subgroups
53
![Page 54: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/54.jpg)
To analyze subgroups, you must recruit subgroups.
• Examine sites and particular clinics for subgroup composition—choose sites accordingly
• Should focus on a few subgroups (or correct for multiple testing in analysis)
• May have sites that are predominately a single minority (Puerto Rico in BSFT was 100% Hispanic)
NOTE: This may create difficulty in identifying subgroup effects separately from site effects in studies with small number of sites
54
![Page 55: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/55.jpg)
Must Protect Integrity of the Overall Study Design
• If subgroup is associated with primary outcome measure, consider stratified randomization (by subgroup to ensure balance across conditions)
• Must decide whether to incorporate subgroups into primary hypothesis testing
• If so, how to incorporate• Fully stratified primary analysis is like running
duplicate trials in each subgroup and would require a large overall sample
55
![Page 56: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/56.jpg)
How much should subgroups be incorporated into primary analysis?• Depends on what subgroup membership affects
—trick is specifying a priori
• Look at 4 possibilities (generally assuming that randomization works) for subgroup effects:– Initial levels differ by subgroup– Initial levels and rates of change differ by subgroup – Initial levels and rates of change differ by subgroup and
failure of randomization within at least one subgroup– Initial levels and rates of change differ by subgroup and
intervention status [i.e., Subgroup X Treatment Interaction]
56
![Page 57: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/57.jpg)
Only Initial Levels Differ
57
• Include a 1df control for subgroup membership– Reduces residual variance – Increases power
• Little cost to include: Minor increase in model complexity—1df (per additional group) if wrong
![Page 58: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/58.jpg)
Initial Level and Rates of Change
58
• 2 df per group (one for intercept and 1 for change-if linear)– Reduces residual variance– Increases Power
• Relatively low cost to complexity and lost df (unless many higher order polynomials in change)
![Page 59: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/59.jpg)
Initial Level & Rates of Change and Randomization Failure in 1 Group
59
• We assume randomization will work—and participants should look similar on average at baseline
• But as you examine more subgroups you increase the chance that one subgroup will have an imbalance (here also differences in slopes)
![Page 60: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/60.jpg)
Initial Levels and Rates of Change Differ by Group and Intervention
60
• Need separate intercepts and slopes for each group BY each intervention condition (have interaction of treatment by subgroup)
• Costly Model (in the sense of statistical power and sample size)– Complicated, many df (wasted if guess wrong)– Basically, fully stratified model—should consider powering within subgroup
![Page 61: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/61.jpg)
61
So, what should you do for your a priori Statistical Analysis Plan?• Mean Differences by subgroup: add 1 df
control for subgroup membership—low cost if wrong
• Different trajectories by subgroup (but = intervention effect)– With existing evidence: plan to include
subgroup specific rates of change – No Prior Evidence:
• Could include examination of trajectories blind by condition
• May be better to test control group for differences in trajectories by subgroup
![Page 62: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/62.jpg)
So, what should you do for your a priori Statistical Analysis Plan? (cont.)• Different intervention effects—Full
stratification with test for intervention interaction effects—Large Trial– Note that Testing for interaction does not
“solve” the problem
– Must plan for possibility that interaction is significant—then would want power to show effect within a subgroup
62
![Page 63: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/63.jpg)
How many do you need in a subgroup if want to explore?
• If subgroups are not main emphasis of trial, difficult to power subgroup effects– Emphasize effect sizes and minimal sample to get
stable estimates– Nevertheless, useful to have a feel for power
• Will look at power implications– For mean (initial) level differences across subgroups– For rate of change differences across subgroups
63
![Page 64: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/64.jpg)
64
N per group to Have 80% Power For Simple Mean Differenceat Point in Time
Sta
ndar
dize
d D
iffer
ence
Sample Size Per Group
0.1
0.3
0.5
0.7
0.9
0 100 200 300 400 500
Source: PASS 2005 Assumes normality and 2 group comparison
![Page 65: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/65.jpg)
Power for Differences in Slopes• Assumes 4 assessment times
• Lose 15% at T2, 5% more at T3 & T4
• Compound symmetry in Errors=.20
• Random effects in intercept and slope terms
• Effect size is the standardized mean difference at the LAST time (assumes largest effect at last observation)
• Uses RMASS program from Don Hedeker– Hedeker, Gibbons, & Waternaux, 1999 JEBS– Available at http://tigger.uic.edu/~hedeker/
65
![Page 66: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/66.jpg)
66
Sample Size per Subgroup for 80% PowerGrowth Curve Showing Mean Difference at Last of 4 Times
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0 50 100 150 200 250 300 350
n per group
Effe
ct S
ize
Linear Growth CurveQuadratic Growth Curve
Assumes 2 group comparison
![Page 67: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/67.jpg)
Note About Power Statements• These are for 2 group comparisons
• If you want to find power for testing difference in intervention effects across 2 (or more) groups (an interaction effect) would need to simulate if planning on a growth curve framework with missing data
• Rough, overestimate—4 times the per group n—should over-power the interaction effect
• However, if want power within subgroups—4 times per group n is the proper sample size
67
![Page 68: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/68.jpg)
Potential Difficulties with Subgroup Analyses• Disentangling subgroup effects from site
effects
• Interpretation of subgroup findings– Are measurement instruments equally valid
and reliable across subgroups (need at least 100 per group even for small invariance analysis)
68
![Page 69: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/69.jpg)
Potential Difficulties with Subgroup Analyses (cont.)• Interpretation of subgroup findings (cont.)
– What does group membership mean? Proxy for?• What person-specific variables do we need to
understand subgroup difference• What contextual variables do we need?
– Local, neighborhood and regional factors (zip codes of individuals and clinics?)
– Treatment context and other site level factors
69
![Page 70: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/70.jpg)
Summary of Issues with Secondary Analysis• Analyses that use treatment assignment as the
only predictor*:– Have a strong causal interpretation– Are useful to characterize the full impact of an
intervention—substance treatment has multiple targets
– But too many secondary outcomes may cause a Type I error, unless a multiple testing procedure is followed
– Need to be clear about the number of secondary outcomes considered
70* Could also include other factors measured at baseline as control variables
![Page 71: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/71.jpg)
Summary of Issues with Secondary Analysis (cont.)• All other secondary analyses (mediation,
associational, and subgroup):– Cannot be interpreted as causal without strong
assumptions
– Need to assess for confounding variables
– Be careful not to over-interpret—Cannot rule out unobserved confounding
– Are also susceptible to Type 1 errors—• limit the number of analysis and/or incorporate a multiple
testing strategy and be clear in presentation of what you have done
71
![Page 72: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/72.jpg)
Suggestions• All manuscripts should have an analysis plan
pre-specified
• Any findings that come from exploratory analyses need to be clearly designated in the manuscript
• May be useful to cite the CTN dissemination library for the particular trial (for readers reference for the other different outcomes that have been published on the same data)
72
![Page 73: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/73.jpg)
73
WRAPPING IT UP!Highlights
![Page 74: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/74.jpg)
74
Summary• Secondary analyses are important and necessary
components of clinical trial research
• Secondary outcomes and analyses should be considered and identified during protocol and/or SAP development
• Interpret results appropriately: exploratory, confirmatory, descriptive, not causal
• Always report whether analysis/outcome was pre-specified or post hoc and the number performed
![Page 75: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/75.jpg)
75
Summary (cont.)• CTN encourages multiple analyses & publications
• Describe methods clearly and accurately
• It is essential to acknowledge the limitations
• Beware of:− Excessive testing− Data dredging− Misinterpreting correlations as causes− Confounding variables
![Page 76: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/76.jpg)
76
Q&A – Questions / Comments
Alternatively, questions can be directed to the presenter by sending an email to [email protected].
![Page 77: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/77.jpg)
77
References• Hien, et al. 2009. J Consult Clin Psychol, 77(4); 607–619.
• Lagakos, S. 2006. The challenge of subgroup analysis—Reporting without distorting. NEJM, 354; 16.
• Nunes, et al. 2011. American Journal of Drug and Alcohol Abuse, 37; 446-452.
• Wang, et al. 2007. Statistics in medicine – Reporting of subgroup analyses in clinical trials. NEJM, 357; 21.
![Page 78: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/78.jpg)
78
Survey ReminderThe NIDA CCC encourages all to complete the survey issued to participants directly following this webinar session, as this is the primary collective tool for rating your experience with this and other webinars, and for communicating the interests and needs of CTN members and associates.
See you all in 2014!
![Page 79: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/79.jpg)
79
A copy of this presentation will be available electronically after this session.
http://ctndisseminationlibrary.org
![Page 80: SECONDARY ANALYSES IN CLINICAL TRIALS](https://reader035.vdocument.in/reader035/viewer/2022062501/56816695550346895dda78bc/html5/thumbnails/80.jpg)
80
THANK YOU FOR YOUR PARTICIPATION