Comparing Research Designs

Patrick BarlowStatistical and Research Design Consultant, Graduate School

of Medicine, UTKPhD Candidate in Evaluation, Statistics, and Measurement,

UTK

On the Agenda

Important considerations in research design Reliability & validity Biases & confounding Strength of evidence

Observational Research Designs Cross-sectional study Case-control study Cohort study

Experimental Research Designs The Basics of Factorial and Crossover Trials

Important considerations

in research designConfoundin

gBias

ReliabilityValidity

Reliability & Validity

Reliability Validity Refers to the consistency of

an instrument/measurement.

Thought of as an individual’s “true score” on the phenomenon you aim to measure minus “measurement error”

Two common types of reliability Internal consistency:

Cronbach’s alpha, KR20 Inter-Rater: Kappa

statistic

Necessary but not sufficient in determining validity.

Refers to the accuracy of an instrument/measurement

In other words, “the degree to which you’re measuring what you claim to measure”

Two broad types of validity Internal validity External validity

Internal vs. External Validity

One of the strengths of randomized designs are that they have substantially higher internal & external validity.

Internal Validity: refers to the integrity of the experiment itself. It is the ability to draw a causal link between your treatment and the dependent variable of interest.

External Validity: refers to the ability to generalize your study findings to the population at large. In other words, are your findings from a sample of UTMCK patients with HTN going to apply to all patients with HTN?

Threats to Internal Validity

Concerns the accuracy of measurement within the study

Shadish, Cook & Campbell (2002) summarized a number of possible threats to internal validity, which can severely jeopardize the findings of a study. In particular: History, Mortality, & Maturation Repeated Testing Confounding Diffusion & Compensatory Rivalry

Threats to Internal Validity

Diffusion & Compensatory Rivalry Diffusion: Treatment effects can “spill over” or “spread” across

treatment groups. EX: Patients from different groups live near each other and discuss / share their experiences or treatments.

Compensatory Rivalry: Patients perform in a certain way because they know they’re in the control / experimental groups.

Threats to Internal Validity

History, Mortality, & Maturation History: events external to the experiment influence the

participants’. EX: Superstorm Sandy hits during a crossover trial in New Jersey.

Mortality: Patients either die (mortality) or drop out of the study (attrition) at different rates.

Maturation: Patients change over the course of the treatment, which influences results. EX: Children grow up during the course of a pediatric clinical trial.

Repeated Testing Patients can become “test-wise” if given the same subjective

test multiple times, or they become conditioned to being tested (EX: patient’s pulse increases before a needle stick).

External Validity

The ability to generalize the findings of your study to the relevant population.

Threatened by Bias Confounding Non-experimental design (i.e. case-control vs. RCT) Lack of randomization

External validity is the strongest when a true experimental design is used.

Confounding

A confounder is a variable that is causally associated with the outcome (DV) and may or may not be causally associated with the exposure (IV)

Causes spurious conclusions & inferences to be made about a set of variables

Reduced through Randomization Matching Statistically controlling (covariates)

Confounding Example

Smoking Hx

HPV

Cervical Cancer

?

Bias in Research

The result of systematic error in the design or conduct of a study

Can artificially “trend” results Toward the Null hypothesis Toward the Alternative

hypothesis

A major problem to consider when planning any study

Common Biases

Selection bias: one relevant group in the population (e.g. cases positive for predictor variable) has a higher probability of being included in the sampleMisclassification can be either unsystematic

(random) or systematic (bad)

Information: bias from erroneously classifying people in exposure/outcome categoriesRecall/Response: bias associated with

inaccurate recall of exposure or representation of true exposure (self-report)

Experimenter/Interviewer bias: Differential treatment of participants in treatment and control groups

Publication: the tendency to publish only “positive” or “significant” findings.

Strength of EvidenceThe Bradford Hill Criteria

Provides researchers with seven criteria for assessing strength of evidence.

Strength of association (i.e. effect size) Consistency (i.e. reliability) Specificity Temporal relationship Biological gradient Plausibility Coherence Experiment (reversibility) Analogy (consideration of alternate explanations)

Pyramid of Clinical Evidence

RCTCohort Studies

Case Control Studies

Case Series

Case Reports

Ideas, Editorials, Opinions

Animal research

In vitro (‘test tube’) research

Systematic Reviews & Meta-

analyses

Evidence Summaries

Level 2 Evidence

Level 1 Evidence

Level 3 Evidence

Cross-Sectional Studies: Level

2.3

Observational Research Designs

Cross-sectional

Case-controlCohort

Cross-Sectional Studies

“Snapshot” of a population.

People are studied at a “point” in time, without follow-up.

Strength of evidence…

What are some research questions that can be answered with cross-sectional designs?

Advantages and Disadvantages of Cross-

Sectional StudiesAdvantages Disadvantages Fast and inexpensive No loss to follow-up Springboard to

expand/inform research question

Can target a larger sample size

Can’t determine causal relationship

Impractical for rare diseases

“Garbage in, garbage out”

Risk for nonresponse

Case-Control Studies

Always retrospective Prevalence vs. Incidence

A sample with the disease from a population is selected (cases).

A sample without the disease from a population is selected (controls).

Groups are compared using possible predictors of the disease state.

Advantages and Disadvantages of Case-Control

StudiesAdvantages Disadvantages

High information yield with few participants

Useful for rare outcomes

Cannot estimate incidence of disease

Limited outcomes can be studied

Highly susceptible to biases

Strategies for Sampling Controls

Population versus hospital/clinic-based controls

Matching Individual level Group level

Using two or more control groups

Cohort Studies

A “cohort” is a group of individuals who are followed or traced over a period of time.

A cohort study analyzes an exposure/disease relationship within the entire cohort.

Groups selected based on exposure to a risk factor.

Level of evidence?

Cohort Design

Prospective vs. Retrospective Cohort

StudiesExposure Outcome

Prospective

Assessed at the beginning of the study (present)

Followed into the future for outcome

Retrospective

Assessed at some point in the past

Outcome has already occurred

Advantages and Disadvantages of Cohort

StudiesAdvantages Disadvantages

Establish population-based

incidence

Temporal relationship inferred

Time-to-event analysis

possible

Used when randomization not

possible

Reduces biases (selection,

information)

Lengthy and costly

Not suitable for rare/long-

latency diseases

May require very large

samples

Nonresponse, migration and

loss-to-follow-up

Sampling, ascertainment and

observer biases

Experimental Designs

The Basics of Factorial and Cross-Over Designs

Experimental DesignsWhat are They?

Considered to be the “gold standard” of clinical

evidence because:

Randomization is used to reduce the effect of biases

and confounding variables

Patients (single) and researchers (double) can be

blinded to the intervention

High internal and external validity allow for assessing

cause and effect relationships.

The most basic experimental design is a

“Parallel trial.”

Patients are randomized into one of two groups, and

remain in the same group throughout the study.

“Double-blind trials”

Factorial DesignsWhat are They?

Factorial designs allow for researchers to test multiple interventions or treatment combinations in a single study. For example: drug A or Drug B and 3x per

week or everyday dose cycle.

The simplest form of this design is a 2x2 factorial design.

Allows researchers to test individual treatment effects and/or interactions between different treatments.

Looks like a “grid”

Factorial DesignsWhy are They Used?

Factorial design are commonly used to effectively test multiple treatments or “Main effects” in a single study. More efficient and more statistically powerful than multiple single

intervention studies

Especially useful for testing interactions among different interventions or treatments

Main Effects

Interactions

Factorial DesignsExample

Dose Cycle

StatinRosuvastat

in (Crestor)

Atorvastatin (Lipitor)

3x Per Week

M LDL M LDL

Everyday M LDL M LDL

What is the effect of dose (3x pw or everyday) and statin (Rusuvastatin or Atorvastatin) regimen on mean LDL Cholesterol?

Cross-over DesignsWhat are They?

A cross-over trial design involves giving the two or more interventions/treatments to a single group of patients.

At its most basic, this trial tests the efficacy of two treatments where each patient spends a period of time under both treatment options.

Patients are randomized into which treatment they receive first, and then swap to the other treatment after a predetermined time.

Cross-over DesignsWhat are They?

A

B“Cross-over”

A

B

Cross-Over DesignsWhy are They Used?

Cross-over trials are useful because they reduce confounding factors associated with between-subjects designs. Patients serve as their own controls Useful for time-dependent research questions Higher statistical power than between subjects designs due to no

between-subjects error (i.e. need less patients to find statistical significance).

Cross-Over DesignsExample

3x Per Week

Treatment

Everyday Treatmen

t

Everyday Treatme

nt

3x Per Week

Treatment

Week Six

Disadvantages of RCT Designs

Extremely time and resource demanding

Unethical in many situations

Poor external validity if the RCT is too highly controlled

Difficult to study rare events

Therapeutic misconception

In Pairs…

Work together to brainstorm an example of how your topic could be addressed using 1) a Cross-Sectional design, 2) a case-control design, 3) a prospective or retrospective cohort design, and an RCT (Parallel, factorial, or cross-over).

Be prepared to share your responses