Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Jonathan HallettEvidence + Effectiveness in Health PromotionEVALUATION DESIGNS

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Lecture overview

Evaluation designs

Internal and external validity

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Evaluation designs

There is no single, perfect evaluation design that is best for all health promotion programs, or even for a specific type of health promotion evaluation task

“...it all depends”ObjectivesPracticalities$$$$

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Evaluation types

Cross sectional – measures the prevalence of health outcomes and/or determinants of health, in a population at a specific point in time or over a short period

Retrospective case control – begins with absence or presence of an outcome/disease and looks backward to detect causes

Longitudinal (cohort) – people (subjects) are followed over time with continuous or repeated monitoring of risk factors or health outcomes

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Study designs

X = intervention

O = observation/measurement

Group 1 O1 X1 O2 X2 O3 O4

Group 2 O1 O2 O3 O4

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 1

Single Group, Post-Test only

X___________O Problems

Did a change really occur?What did people start at?Was the change due to your program?AdvantageQuick, not attrition, cheap

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 2

Single Group, Pre-test & Post-test

O1___________X___________ O2

AdvantageAbility to detect if a change has occurred

ProblemWas the change because of your intervention/ program?

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 3

Non-Equivalent Comparison Groups Post-test only

X________________O ________________O

Advantage Avoids effect of pre-measurement (test effects), i.e. can be an

intervention itself Problem

Was the change because of your intervention? Were both groups the same when the program began? Comparison group may be different e.g. fitter, older etc

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 4

Non-equivalent Comparison Groups - Pre-test & Post-test

O1 _________X____________ O2

O1 ______________________ O2

Advantage:Compare results of people who do and do not receive the program

Know what groups have changed from (i.e. were they the same at the start)

(This is a stronger design but ...)

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 4, continued...

Problem- Similar to design 3, unsure if change was your intervention

- May need to control for known differences between groups (strengthen design)

- Cost

- Finding appropriate comparison group

- Contamination of comparison group

- Extent to which program can be generalised from one group to another

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 5

Single Group Time Series

O1____O2_____O3___X___O4____O5____O6

AdvantageObserve natural changes (size & direction) before & after programBest used for program subject to seasonal / cyclical changes

ProblemTesting effectCost

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Design 6

Randomised Control Trial (Equivalent Groups, Pre/Post test)

O1 __________X__________ O2

O1 ______________________ O2

Rules out alternative explanations Cost? Strong design

Population Population RR

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Randomised Control Trials

Representative sample of target group with all having equal chance of being selected

Random assignment to experimental or control groups

Implementation of health promotion intervention

Unexposed group for comparison

Prospective

One or more pre-tests

CLUSTER randomised trial: e.g. Schools/workplaces - randomisation takes place at

organisational level.

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Quasi Experimental Design

Designs 1 – 5

Can use a comparison group but not randomised e.g. State campaign compared to another State

Time series studies with tests taken prior to the intervention can observe a trend

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Experimental designs

Most widely acknowledged scientifically

RCT (individuals)

Cluster RCT (communities)

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Other designs

Staggered Designs

O1_____X1____O2

O1_____X1____O2

Cross-over designs

O1_____ X1____O2 __________O3

O1___________O2 ____ X1 ____O3

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Study biases

Self-selection – may be more ready to change

Participant retention (control groups can adjust for this)

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Cohort versus Waves

Cohort – follow the same people - Each has a unique identifier

- Can see changes within people

Waves (monitoring) – cross-sectional measures of population

- New sample at each time point

- See trends within populations

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Which evaluation design?

Some are better than others

Which design produces better evidence that your program has worked?

Choosing a design will depend on what information you want, what you need and what you are practically able to get given resources and time constraints

Usually, as a minimum, aim for a single group, pre-test & post-test design (Design 2)

Evidence + Effectiveness in Health Promotion

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Next week

Data collection methods

Quantitative, qualitative and mixed methods

Evidence + Effectiveness in Health Promotion