swl 579a session 5 methodological challenges in prevention science guest lecturer: eric brown, ph.d....

SWL 579A Session 5SWL 579A Session 5

Methodological challenges Methodological challenges in prevention sciencein prevention science

Guest Lecturer: Eric Brown, Ph.D.Guest Lecturer: Eric Brown, Ph.D.School of Social WorkSchool of Social WorkUniversity of WashingtonUniversity of Washington10/28/0910/28/09

Intervention Research

Programs originating in practice (e.g., Homebuilders, Fountain House), those originating from intervention researchers (e.g., MC, YM), or a combination (e.g., CTC)

Usually conducted by program designer but sometimes involves independent evaluator (e.g., Mathematica Policy Research)

Challenges in Intervention Research

2.70

2.75

2.80

2.85

2.90

2.95

3.00

3.05

3.10

13 14 15 16 17 18Age

Leve

l of Sc

hool

Bon

ding Full Treatment

Late TreatmentControl

Effects of SSDP Intervention Effects of SSDP Intervention on School Bonding from Age on School Bonding from Age

13 to 1813 to 18

Hawkins, Guo, Hill, Battin-Pearson & Abbott (2001)

Methodological challenges in Methodological challenges in prevention scienceprevention science

Unit of analysis.Unit of analysis. Measurement issues.Measurement issues. Heterogeneity of effects in Heterogeneity of effects in

different subgroups.different subgroups. Assessing intervention effects on Assessing intervention effects on

developmental change. developmental change. Attrition and missing data.Attrition and missing data.

3 Design Stages3 Design Stages

Pre-Intervention Assignment Pre-Intervention Assignment DesignDesign

Intervention DesignIntervention Design

Post-Intervention DesignPost-Intervention Design

Intervention AssignmentIntervention Assignment

RandomizationRandomization

Balance, Matching, BlockingBalance, Matching, Blocking

Cluster Random Assignment Cluster Random Assignment

What are the Fatal Design What are the Fatal Design Flaws in a Trial?Flaws in a Trial?

Pre-Intervention Assignment:Pre-Intervention Assignment:– Extreme Selection BiasExtreme Selection Bias– Not a Large Enough Sample is DrawnNot a Large Enough Sample is Drawn

Intervention:Intervention:– Contamination/LeakageContamination/Leakage– Participation BiasParticipation Bias

ImplementationImplementation ParticipationParticipation Adherence Adherence DosageDosage

– Drop-out during intervention periodDrop-out during intervention period

Intervention DesignIntervention Design

Intervention and control Subjects Intervention and control Subjects are differentare different

ContaminationContamination

Randomized at wrong levelRandomized at wrong level

Low intervention deliveryLow intervention delivery

Large drop-outsLarge drop-outs

Post-Intervention DesignPost-Intervention Design

Large attritionLarge attrition

Differential attritionDifferential attrition

Differential measurement errorDifferential measurement error

When it’s not possible to When it’s not possible to randomize or when randomize or when randomization fails…randomization fails… Propensity score = Probability of receiving Propensity score = Probability of receiving

intervention given observed covariates.intervention given observed covariates. Often estimated using logistic regression.Often estimated using logistic regression. Discriminates between experimental and Discriminates between experimental and

control groups.control groups. Can be thought of as a “balancing score”Can be thought of as a “balancing score”

– Within groups with similar propensity scores, Within groups with similar propensity scores, distribution of covariates will be similar across distribution of covariates will be similar across experimental and control groups.experimental and control groups.

– Allows post hoc matching based on propensity Allows post hoc matching based on propensity score instead of all covariates directly.score instead of all covariates directly.

Propensity Scores (continued)Propensity Scores (continued)

Can also be used when follow-up of all Can also be used when follow-up of all participants is not possible.participants is not possible.

Reduces non-intervention related Reduces non-intervention related differences between experimental and differences between experimental and control groups.control groups.

Gives better estimates of intervention Gives better estimates of intervention effects (reduced bias).effects (reduced bias).

Statistical PowerStatistical Power

HypothesesHypotheses::

HH00: : μμ11 = = μμ22

HHAA: : μμ11 ≠ ≠ μμ22

Type I error rate (Type I error rate (αα): probability of rejecting H): probability of rejecting H00 when TRUE when TRUEType II error rate (Type II error rate (ββ): probability of accepting H): probability of accepting H00 when FALSE when FALSE

ExamplesExamples::

HH00: Unsafe to cross the street: Unsafe to cross the streetHHAA: Safe to cross the street : Safe to cross the street

HH00: The defendant is innocent: The defendant is innocentHHAA: The defendant is guilty: The defendant is guilty

Power = 1 – β: probability of rejecting HPower = 1 – β: probability of rejecting H00 when FALSE. when FALSE.

Strategies to Increase Strategies to Increase PowerPower

Increase sample sizeIncrease sample size

Balance, then randomizeBalance, then randomize

Use multiple outcome measuresUse multiple outcome measures

Draw more homogeneous sampleDraw more homogeneous sample

Analyze at multiple (appropriate) Analyze at multiple (appropriate) levelslevels

ExampleExample::The Community Youth The Community Youth Development Study (CYDS)Development Study (CYDS)

A randomized controlled trial to A randomized controlled trial to test the effectiveness of the test the effectiveness of the Communities that Care Communities that Care prevention operating system.prevention operating system.

To foster healthy youth To foster healthy youth development in communities:development in communities:

Reduce levels of riskReduce levels of risk

Increase levels of promotion and Increase levels of promotion and protection protection

Reduce levels of youth substance use, Reduce levels of youth substance use, violence, and other problem behaviorsviolence, and other problem behaviors

Communities That CareCommunities That Care

Community Youth Community Youth Development Study (CYDS)Development Study (CYDS)

CYDS will test if CTC increases CYDS will test if CTC increases positive youth development in positive youth development in communities. communities.

CYDS Research QuestionsCYDS Research Questions

Does CTC improve community Does CTC improve community planning and decision making? planning and decision making? ((Process OutcomesProcess Outcomes))

Does full installation of CTC affect Does full installation of CTC affect targeted risk and protective targeted risk and protective factors and healthy or problem factors and healthy or problem behaviors? (behaviors? (Behavior OutcomesBehavior Outcomes))

Adoption of Science-based

Approaches

CollaborationAppropriate Prevention Program Selection and

Implementation

Positive Youth Development

Decreased Risk and Enhanced Protection

CTC Implementationand Technical Assistance

Community Norms

Social Development Strategy

Community Support

System TransformationConstructs

System OutcomesSystem Catalyst

Theory of Change for Communities That Care Prevention System Transformation

CYDS Design: Community CYDS Design: Community Selection and MatchingSelection and Matching

Forty one free standing incorporated Forty one free standing incorporated towns of less than 100,000 population towns of less than 100,000 population were chosen and matched in 1997.were chosen and matched in 1997.

The community pairs are similar in:The community pairs are similar in:– Population SizePopulation Size– Demographic DiversityDemographic Diversity– Crime StatisticsCrime Statistics– Socioeconomic composition Socioeconomic composition – Drug useDrug use

CYDS Design (cont.)CYDS Design (cont.)

The Diffusion Project (1997-2002) found that in 13 The Diffusion Project (1997-2002) found that in 13 community pairs, neither community was using community pairs, neither community was using prevention science to guide its drug abuse prevention science to guide its drug abuse prevention efforts.prevention efforts.

12 sets of these paired communities agreed to be 12 sets of these paired communities agreed to be

randomly assigned to CTC or control condition in randomly assigned to CTC or control condition in 2003. 2003.

STUDY DESIGNSTUDY DESIGN

Randomized Controlled Trial

2003-2008

Randomize

5-Year Baseline

1997-2002

98 99 ‘00 ‘01 ‘02

CKICRD

2003 2004 2005 2006 2007 2008

Control

Intervention

CTCYS

CKICRD

CKICRD

CKICRD

CKICRD

YDS YDS YDS

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCYS: Cross-sectional student survey of 6th-, 8th-, 10th-, and 12th-grade students using the CTC Youth SurveyCKI: Community Key Informant InterviewCRD: Community Resource Documentation measuring effective prevention programs and policies in the community CTC Board: CTC Board Member InterviewYDS: Longitudinal Youth Development Survey of students in the class of 2011 starting in 5th grade in spring 2004

Planning Implement selected interventions

CTCYS CTCYS

CTCYS CTCYS CTCYS

CTCYS CTCYS

CTCYS

CKICRD

YDSYDS

YDS YDSYDSYDSYDS

Measurement ToolsMeasurement Tools CTC Team Member InterviewsCTC Team Member Interviews to to

measure the CTC process in each measure the CTC process in each communitycommunity

Community Key Informant InterviewsCommunity Key Informant Interviews to to measure adoption of prevention science measure adoption of prevention science as a planning framework for preventive as a planning framework for preventive action in communitiesaction in communities

Community Resource DocumentationCommunity Resource Documentation system to assess location and reach of system to assess location and reach of programs consistent with proven programs consistent with proven prevention approachesprevention approaches

Student surveysStudent surveys to measure risk and to measure risk and protection and youth problem behaviorsprotection and youth problem behaviors



2003-2008

Randomize

5-Year Baseline

1997-2002

98 99 ‘00 ‘01 ‘02

CKICRD

2003 2004 2005 2006 2007 2008

Control

Intervention

CTCYS

CKICRD

CKICRD

CKICRD

CKICRD

YDS YDS YDS

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d



CTCYS CTCYS

CTCYS CTCYS CTCYS

CTCYS CTCYS

CTCYS

CKICRD

YDSYDS

YDS YDSYDSYDSYDS

Repeated Cross-Sectional Repeated Cross-Sectional Youth Surveys - CTC SurveyYouth Surveys - CTC Survey

Target samples include all 6th, 8th, 10th, Target samples include all 6th, 8th, 10th, and 12th grade public school students in and 12th grade public school students in each community (Total N’s range from each community (Total N’s range from 160 - 2000 students per community)160 - 2000 students per community)

Used to prioritize specific risk and Used to prioritize specific risk and protective factors for attentionprotective factors for attention

Provide data on population trends in risk, Provide data on population trends in risk, protection, and outcomes in each protection, and outcomes in each community from 1998 to 2008community from 1998 to 2008



2003-2008

Randomize

5-Year Baseline

1997-2002

98 99 ‘00 ‘01 ‘02

CKICRD

2003 2004 2005 2006 2007 2008

Control

Intervention

CTCYS

CKICRD

CKICRD

CKICRD

CKICRD

YDS YDS YDS

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d



CTCYS CTCYS

CTCYS CTCYS CTCYS

CTCYS CTCYS

CTCYS

CKICRD

YDSYDS

YDS YDSYDSYDSYDS

Longitudinal Youth Surveys -Longitudinal Youth Surveys -Youth Development Study (YDS)Youth Development Study (YDS)

Target samples include all 5th grade Target samples include all 5th grade public school students in each community public school students in each community recruited in 2003 and 2004recruited in 2003 and 2004

Provide data on individual level changes Provide data on individual level changes in risk, protection, and outcomes from 5th in risk, protection, and outcomes from 5th through 9th grades in each communitythrough 9th grades in each community

Provide data on individual students’ Provide data on individual students’ exposure to prevention activities in each exposure to prevention activities in each communitycommunity

Panel-Panel-Youth Development Survey Youth Development Survey (YDS)(YDS)

Annual survey of panel recruited from the Class Annual survey of panel recruited from the Class of 2011 (5of 2011 (5thth grade in 2004) grade in 2004)

Active, written parental consentActive, written parental consent



2003-2008

Randomize

5-Year Baseline

1997-2002

98 99 ‘00 ‘01 ‘02

CKICRD

2003 2004 2005 2006 2007 2008

Control

Intervention

CTCYS

CKICRD

CKICRD

CKICRD

CKICRD

YDS YDS YDS

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d

CTCBoar

d



CTCYS CTCYS

CTCYS CTCYS CTCYS

CTCYS CTCYS

CTCYS

CKICRD

YDSYDS

YDS YDSYDSYDSYDS

Youth Development Survey Youth Development Survey (YDS)(YDS)

Participants recruited in grades 5 and 6.Participants recruited in grades 5 and 6. Final consent rate = 76.4%Final consent rate = 76.4%

Sixth Sixth GradeGrade

Eligible Eligible PopulatioPopulatio

nn

Percent Percent ConsenteConsente

dd

Percent Percent SurveyeSurveye

dd

Total Total SurveyeSurveye

dd

ExperimentExperimentalal

31703170 76.2%76.2% 75.4%75.4% 23912391

ControlControl 26212621 76.7%76.7% 76.3%76.3% 19991999

TotalTotal 57915791 76.4%76.4% 75.8%75.8% 43904390

2007 YDS2007 YDS

88thth Grade Grade Eligible Eligible PopulatioPopulatio

nn

Percent Percent SurveyedSurveyed

Total Total SurveyedSurveyed

ExperimentalExperimental 24062406 95.6%95.6% 23002300

ControlControl 20012001 96.9%96.9% 19401940

TotalTotal 44074407 96.2%96.2% 42404240

96.2% Overall Student Participation96.2% Overall Student Participation 11.9% (n=525) have moved out of project 11.9% (n=525) have moved out of project

schoolsschools

The CONSORT (Consolidated The CONSORT (Consolidated Standards of Reporting Trials) Standards of Reporting Trials) Statement… Statement…

was developed by a group of clinical was developed by a group of clinical trialists, biostatisticians, trialists, biostatisticians, epidemiologists and biomedical epidemiologists and biomedical editors as a means to improve the editors as a means to improve the quality of reports of randomized quality of reports of randomized controlled trials (RCTs). controlled trials (RCTs).

CONSORT Flow DiagramCONSORT Flow Diagram

……shows the progress of shows the progress of participants throughout a RCT. participants throughout a RCT.

Thoma et al., 2006

1346 students in grade 5 (67.2%) 1987 students in grade 6 (99.3%) 1921 students in grade 7 (95.0%) 1910 students in grade 8 (95.4%)

1867 students in grade 5 (77.6%) 2368 students in grade 6 (98.5%) 2274 students in grade 7 (94.6%) 2272 students in grade 8 (94.5%)

12 communities assigned to

INTERVENTION condition

12 communities included in analysis

2 communities(1 matched pair)

not recruited

12 communitiesassigned to CONTROLcondition

12 communities included in analysis

24 communitiesrandomized

(within 12 matched pairs)

3170 students eligible to participate in panel study

2621 students eligible to participate in panel study

2405 (76.2%) students consented

2002 (76.7%) students consented

41 communitiesin 7 states assessed for eligibility

26 communities(13 matched pairs)

eligible

24 communities(12 matched pairs)

recruited

15 communitiesineligible

186 students did not consent

154 students did not consent

2391 students surveyed in grade 6 (99.4%)








Students who did not meet validity screen were excluded from analysis:

Students who did not meet validity screen were excluded from analysis:

9 students in grade 523 students in grade 6

15 students in grade 512 students in grade 620 students in grade 730 students in grade 8

24 students in grade 728 students in grade 8

Final Analysis Sample: Final Analysis Sample:

CYDS Youth Development CYDS Youth Development Study (panel sample) Grade 7 Study (panel sample) Grade 7

CONSORT flow diagramCONSORT flow diagram

Hawkins et al., 2009

Unit of analysisUnit of analysis

• What is the unit of analysis in your study?What is the unit of analysis in your study?

• Are there multiple units of analysis in your study?Are there multiple units of analysis in your study?

• Does the unit(s) of analysis in your study Does the unit(s) of analysis in your study correspond to your theory of changcorrespond to your theory of change?

• Does the unit of analysis in your study correspond Does the unit of analysis in your study correspond to the unit of randomization?to the unit of randomization?

• Do you have enough units to do the appropriate Do you have enough units to do the appropriate statistical analysis? …to have sufficient statistical statistical analysis? …to have sufficient statistical power?power?

Three-level Pre-post ANCOVA Model

Level 1 (Studenti):

G8Alc30ijk = π0jk + π1jk(G5Alc30ijk) + π1jk(Ageijk) + π2jk(Genderijk) + π3jk(Whiteijk) + eijk

Level 2 (Communityj):

π0jk = β00k + β01k(Intervention Statusjk) + β02k(Populationjk) + β01k(PctFRLjk) + r0jk

Level 3 (Matched-Pairk):

β00k = γ000 + u00k

Four-level (Latent) Growth Model

Level 1 (Time t):

Alc30tijk = π0ijk + π1ijk(Timetijk) + etijk

Level 2 (Student i):

π0ijk = β00jk + β01jk(Ageijk) + β02jk(Sexijk) + β03jk(Whiteijk) + β04jk(Hispanicijk) + r0ijk

π1ijk = β10jk + β11jk(Ageijk) + β12jk(Sexijk) + β13jk(Whiteijk) + β14jk(Hispanicijk) + r1ijk

Level 3 (Community j):

β00jk = γ000k + γ001k(Intervention Statusjk) + γ002k(Populationjk) + γ003k(PctFRLjk) + u00jk

β10jk = γ100k + γ101k(Intervention Statusjk) + γ102k(Populaitonjk) + γ103k(PctFRLjk) + u10jk

Level 4 (Community-Matched Pair k):

γ000k = ξ0000 + v000k

Measurement Measurement issuesissues

• How did you select your measures?How did you select your measures?

• Can the variables in your study be measured by a Can the variables in your study be measured by a single item (question)? Or do you need multiple items single item (question)? Or do you need multiple items (questions) to measure the phenomena?(questions) to measure the phenomena?

• Do the response options for the variables that you Do the response options for the variables that you use in your study cover the full range of the use in your study cover the full range of the phenomena? Should you dichotomize a continuous phenomena? Should you dichotomize a continuous variable?variable?

• If you have multiple items, how did you put them If you have multiple items, how did you put them together to measure the construct?together to measure the construct?

Measurement Measurement issuesissues

(continued)(continued)

• Are the measures normally distributed?Are the measures normally distributed?

• Are the variables in your study directly observable Are the variables in your study directly observable (manifest)? Or are they unobservable (latent)?(manifest)? Or are they unobservable (latent)?

• Do your variables measure the same construct Do your variables measure the same construct across different groups or over different time across different groups or over different time periods?periods?

Measurement Measurement issues issues (continued)(continued)

• Are your measures reliable?Are your measures reliable?

Reliability = the consistency of your Reliability = the consistency of your measurement, or the degree to which an measurement, or the degree to which an instrument measures the same way each time it instrument measures the same way each time it is used under the same condition with the same is used under the same condition with the same participants.participants.

Test – RetestTest – Retest

Internal ConsistencyInternal Consistency

Measurement Measurement issues issues (continued)(continued)

• Are your measures valid?Are your measures valid?

Validity = the "best available approximation to the Validity = the "best available approximation to the truth or falsity of a given inference, proposition or truth or falsity of a given inference, proposition or conclusion” (Cook & Campbell, 1979).conclusion” (Cook & Campbell, 1979).

Internal ValidityInternal Validity

External ValidityExternal Validity

Construct ValidityConstruct Validity

Concurrent / Predictive ValidityConcurrent / Predictive Validity

“ “Face” ValidityFace” Validity

4444

ExampleExample: Community Leader : Community Leader Support for PreventionSupport for Prevention

• If you were deciding how to spend money for If you were deciding how to spend money for reducing substance abuse, what percentage reducing substance abuse, what percentage would you allocate to prevention?would you allocate to prevention?

4545

ExampleExample: Stages of Adopting of : Stages of Adopting of Science-Based Approach to PreventionScience-Based Approach to Prevention

Stage 0:Stage 0: No awarenessNo awareness

Stage 1:Stage 1: Awareness of prevention science Awareness of prevention science terminology/conceptsterminology/concepts

Stage 2:Stage 2: Using risk and protection-focused prevention approach Using risk and protection-focused prevention approach as a as a planning strategy.planning strategy.

Stage 3:Stage 3: Incorporation of epidemiological data on risk and Incorporation of epidemiological data on risk and protection in protection in prevention system.prevention system.

Stage 4:Stage 4: Selection and use of tested and effective interventions Selection and use of tested and effective interventions to to address prioritized risk and protective address prioritized risk and protective factors. factors.

Stage 5:Stage 5: Collection and feedback of program process and Collection and feedback of program process and outcome outcome data and adjustment of data and adjustment of interventions based on data.interventions based on data.

4646

ExampleExample: Prevention : Prevention CollaborationCollaboration

1=1=Strongly disagreeStrongly disagree, 2=, 2=Somewhat disagreeSomewhat disagree, 3=, 3=Somewhat agreeSomewhat agree, 4=, 4=Strongly agreeStrongly agree

Item 1Item 1 There is a network of people concerned about prevention There is a network of people concerned about prevention issues who stay in touch with each other.issues who stay in touch with each other.

Item 2Item 2 Community agencies and organizations rarely coordinate Community agencies and organizations rarely coordinate prevention activities.prevention activities.

Item 3Item 3 Community agencies and organizations work together to Community agencies and organizations work together to address [problems with prevention strategies.address [problems with prevention strategies.

Item 4Item 4 Organizations in [COMMUNITY] participate in joint meetings Organizations in [COMMUNITY] participate in joint meetings to address prevention issues.to address prevention issues.

Item 5Item 5 Organizations in [COMMUNITY] share information with each Organizations in [COMMUNITY] share information with each other about prevention issues.other about prevention issues.

Item 6Item 6 Organizations in [COMMUNITY] coordinate prevention Organizations in [COMMUNITY] coordinate prevention strategies.strategies.

Item 7Item 7 Organizations in [COMMUNITY] participate in joint planning Organizations in [COMMUNITY] participate in joint planning and decision making about prevention issues.and decision making about prevention issues.

Item 8Item 8 Organizations in [COMMUNITY] share money or personnel Organizations in [COMMUNITY] share money or personnel when addressing prevention issues.when addressing prevention issues.

Item 9Item 9 In [COMMUNITY], each organization has a clearly defined role In [COMMUNITY], each organization has a clearly defined role in carrying out the community's prevention plan.in carrying out the community's prevention plan.

0 = No use1 = once or twice2 = three to five times3 = six to nine times4 = 10 to 19 times5 = 20 to 39 times6 = 40 or more times

Example: Self-reported frequency of substance use in the

Raising Healthy Children Project (RF Catalano, PI; see Brown et al., 2005 for details)

11

Frequency distributions for alcohol use in past year (total sample)

0102030405060708090

100

0 1 2 3 4 5 6

0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

GRADE 6 GRADE 7 GRADE 8

GRADE 9 GRADE 10

14

Frequency distributions formarijuana use in past year (total sample)

0102030405060708090

100

0 1 2 3 4 5 60

102030405060708090

100

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

0102030405060708090

100

0 1 2 3 4 5 6

GRADE 6 GRADE 7 GRADE 8

GRADE 9 GRADE 10

15

Assessing intervention Assessing intervention effects on developmental effects on developmental changechange

• Are you assessing intervention effects during the Are you assessing intervention effects during the appropriate developmental period?appropriate developmental period?

• How are you measuring “change?” Linearly? By a How are you measuring “change?” Linearly? By a particular growth function?particular growth function?

• Do you have enough measurement occasions (time Do you have enough measurement occasions (time points) to accurately measure change?points) to accurately measure change?

• What are you measuring change in? Incidence? What are you measuring change in? Incidence? Prevalence?Prevalence?

A Latent Variable Model

f

y1

y2

y3

y4

y5

InterventionStatus

Intercept

InterventionStatus

1

2

3

T

0

1

1 01

1 1 21

T -1

…

The Latent Growth Model (LGM)(aka Latent Curve Analysis)

Slope

Alc30G5 Alc30G6 Alc30G7 Alc30t

Grade 7

Intercept Growth Factor 1 Growth Factor 2

Intercept Growth Factor 1

Intervention StatusGender

Grade CohortAntisocial Behavior

Low Income

Growth Factor 2

Grade 7

Grade 8 Grade 9 Grade 10Grade 6

Grade 8 Grade 9 Grade 10Grade 6

Part 1:Substance

Use vs. Nonuse

Part 2:Frequency of Substance Use

Two-part Latent Growth Model (Brown et al., 2005)

Heterogeneity of effects Heterogeneity of effects in different subgroupsin different subgroups

• Does your exploration of effects in different Does your exploration of effects in different subgroups correspond to theory?subgroups correspond to theory?

• Are the subgroups readily identifiable by variables Are the subgroups readily identifiable by variables in your data set? Or are they “latent classes?”in your data set? Or are they “latent classes?”

• Analytic strategiesAnalytic strategies::

subgroup analysissubgroup analysis

intervention effect moderation (e.g., GAM)intervention effect moderation (e.g., GAM)

finite mixture modeling (e.g., LCA, GMM)finite mixture modeling (e.g., LCA, GMM)

model variation in exposure (e.g., CACE)model variation in exposure (e.g., CACE)

0

10

20

30

40

50

12.7 13.6 14.6

Mean Age

Pre

dic

ted

%

Female-Pgm Female-Ctl

Male-Pgm Male-Ctl

Example: Intervention by Gender Interaction for “Had beer/wine/liquor during the past month”

(Raising Healthy Children Project)

Example: Generalized Additive Model for Nonlinear Baseline by Treatment Interactions (Khoo, 2001).

Example: Growth Mixture Model: Intervention Effects of Good Behavior Game on Aggressive, Disruptive

Classroom Behavior (Petras et al., 2008)

Fig. 1. Impact in Cohort 1 males through seventh grade (N= 199).

CohortGender

Low IncomeAntisocial Behv Intervention

Status

CompositeSubstance Use

ExposureClass

2

studyclub

middleschoolretreats

summercamps

familygroup

workshop

at-homeservices

familyboostersessions

Hypothetical model for multi-component exposure for RHC Project

a a a a a a

b

d

e

f

c

Mixture Randomized Trial Modelusing Complier-Average Causal Effect (CACE) estimation

Attrition and Missing DataAttrition and Missing Data

• Differential attrition (“mortality”) = Differential attrition (“mortality”) = differentialdifferential loss of loss of participants between intervention-control groups.participants between intervention-control groups.

• Threat to…Threat to…

reliability or validity?reliability or validity?

internal or external validity?internal or external validity?

• What to do? First, compare those who remain What to do? First, compare those who remain in the study with those who drop out of the study.in the study with those who drop out of the study.

Attrition and Missing DataAttrition and Missing Data(continued)(continued)

• Methods to deal with missing data:Methods to deal with missing data:

pairwise deletionpairwise deletion

listwise deletionlistwise deletion

hot deck imputationhot deck imputation

regression imputationregression imputation

multiple imputationmultiple imputation

maximum likelihoodmaximum likelihood

SWL 579A Session 5SWL 579A Session 5

Methodological challenges Methodological challenges in prevention sciencein prevention science

Guest Lecturer: Eric Brown, Ph.D.Guest Lecturer: Eric Brown, Ph.D.School of Social WorkSchool of Social WorkUniversity of WashingtonUniversity of Washington10/28/0910/28/09

Lessons Learned• Design content to fit to

the environmental contingencies affecting practice. In schools…– Lesson plan format – Standard Course of

Study– EOG pressures

• Design implementation to fit the organizational context– Who is intervention

agent?– Recruit lead teachers– Teacher team meetings– Limits on copying

• Measure sources of selection bias – cannot recover from unobserved heterogeneity– Assume post

randomization compromises

• Use multiple methods of analysis – traditional covariance control can give wrong answers

• Look for lagged and cumulative effects

Note. Design always trumps statistical adjustments.

swl 579a session 5 methodological challenges in prevention science guest lecturer: eric brown, ph.d....

Documents

intervention effects

different intervention

intervention research

intervention researchers

intervention period

effects of ssdp intervention

control groups

fatal design flaws