psychology 242, dr. mckirnanresearch ethics quasi-experiments studying naturally occurring events ...

Psychology 242, Dr. McKirnan Research Ethics

Quasi-experiments

Studying naturally occurring events

Measurement studies

Retrospective designs

Evaluate existing groups or program

Single shot survey or measure

Non-equivalent groups

Time series designs

Quasi-experimental designsExperimental designs for “studies in nature”.

Back NextHomepage

2Psychology 242Introductionto Research

Psychology 242, Dr. McKirnan Week 12-13, quasi-experimental designs.

True v. quasi-experimental designs

True experiments: Quasi-experiments:

Emphasize internal validity Assess cause & effect (in

relatively artificial environment) Test clear, a priori hypotheses

Emphasize external validity Describe “real” / naturally

occurring events Clear to exploratory hypotheses

Participants assigned to experimental v. control groups

Random or matching Participants & experimenter

Blind to assignment

Existing or non-equivalent groups

Non-random assignment Participants not blind Control group not possible?

Control study procedures Create / manipulate independent

variable Control procedures & measures

Control often not possible May not be able to manipulate

the independent variable Partial control of procedures &

measures

Back NextHomepage



Quasi-experiments: naturally occurring events


Measurement studies





Time series designs

Back NextHomepage



Natural disaster / stressor3-mile island Stress -> immune systemS.F. earthquake Stress & coping

Crime / traumaIraq service, 9 / 11 / 01 PTSD & treatment

Historical event9/11 & air travel ban Contrails & climate change Economic collapse Voting patterns

Publicity / cultural event Info. re: Hormone replacement Health behavior

Measurement studies

1. Naturally occurring events; examples

Event Study question(“Predictor”) (“Outcome”)

Back NextHomepage



Naturally occurring events, 2

Independent variable: Predictor variable (e.g., natural disaster) often assessed after

the event (post-hoc).

Researcher has little control over dose / type of predictor

Participant selection No control over who is exposure to event Some control over selection of sample (e.g., via targeted

sampling)

Many potential confounding variables Outcome (dependent) variables:

No control with archival data Some control with surveys Use retrospective (measured) variables to clarify

interpretation of outcomes or test hypothesis.

Back NextHomepage



Retrospective Event Outcomevariable(s) (“Predictor variable”) variable Social support earthquake [v. control city?] stress & coping

Psych. history crime / trauma mental health[archive? Self-report?] [v. control people?]

Personal attitudes historical event voting patterns

Demographics cultural event health behavior

Naturally occurring events: Retrospective designs

Using retrospective (measured) variables to clarify interpretation of outcomes or test hypothesis.

Back NextHomepage



Quasi-experiments: Existing groups


Measurement studies





Time series designs

Back NextHomepage



Existing groups

Single self-selected group; no comparison possible users of psychotherapy (or any product) members of group or cult [contrast with demographically

matched controls?]

Two or more groups, with self-selection and / or "non-blind" assignment Psychological interventions: therapy v. wait list, etc.

Two or more groups, no random assignment Comparing schools / cities / existing groups…

Existing groups:

Back NextHomepage



May control selection of study group, or must use Convenience sample. Other data may be available about group.

Group

Typically no control over event. Not a true Independent Variable

Naturally occurring event or social change Observe1

Dependent Variable(s):May or may not have control over measures (e.g., surveys v. archival measures).

“One shot” case studies

Typical use: Surveys or measures after an event.

Heuristic value: generating hypotheses for later study or confirm controlled data in “real world” setting.

Internal / External validity:

No control over selection of people into the event.

Potentially no control over selection into measurement group.

No control group; uncontrollable event, or other groups may not “need” the intervention (e.g., therapy)

Back NextHomepage



Example of one-shot case study

Research questions: Does psychotherapy “work” from consumer view? Who gets therapy / what does it consist of? Do consumer responses vary by type of therapy?

Research approach: One shot case study / survey

Sampling frame: Any therapy or psychological service user No real information re: population of therapy users.

Sampling procedure: 4,100 Consumer reports readers responding to “in

magazine” mail-back survey form

Example: Consumer Reports psychotherapy survey[Click for paper]

Back NextHomepage



One shot: Consumer reports survey, 2

Negatives: Selection bias

no control over who got therapy (self-selection)

of those who got therapy, no control over who returned a survey (secondary self-selection)

Cursory outcome measures: satisfaction rather than mental health

Positives: Huge, national sample Wholly anonymous, 3rd party data collection; less bias “Real world” assessment of product quality

Experimental Controls Evaluate by gender, type of treatment, medications, to

provide more differentiated analysis

Back NextHomepage



One shot: Consumer reports survey, 3

Key distinction in psychological interventions: “efficacy” v. “effectiveness” research EfficacyEfficacy; “true” experimental design / Lab-basis

Rigorous controls; High internal validity Test basic theory or highly specific technique

Do the specific ingredients (or theory…) of this treatment validly induce the key outcome?

EffectivenessEffectiveness; quasi-experimental; “natural” or applied setting

Less or no control; naturally occurring treatment

High external validity

Does treatment “work” in real patients w/real therapists?

Back NextHomepage



Comparisons of study types: (Consumer reports 4)

Efficacy experiment v.Efficacy experiment v.• One specific diagnosis • Rigorous control group

• “no treatment” condition • “attention control”

• Random assignment • Manualized / uniform treatments

• High Fidelity to treatment method • Fixed number of sessions.

• Well operationalized outcomes, e.g., • clinician-diagnosed disorder• Standard / validated self-report

symptom scales

• “Blind" raters or diagnosticians ("single-blind“: patient & therapist know what the treatment is..)

• Patients followed for a fixed period

Effectiveness research• Multiple diagnoses & severities • No control group, 2nd controls

• Archival, via pt. characteristics

• Self-selection; “shopping” • Multiple / mixed treatments

• Highly tailored to patient • # sessions is patient based.

• Diverse, self-referenced outcomes • Subjective sense of “wellness” • Lessening of “problem” behaviors

or moods • Personal assessment of functioning

• Self-rated: cannot be "blind"

• Diverse times since treatment• Retrospective rather than

prospective

Back NextHomepage



Consumer reports survey, 5.

Survey findings on therapy effectiveness: People who got more treatment (> 6 months) did better.

For general ψ health MH specialists did best, marriage counselors worst.

For patients’ presenting problem(s) all specialists did about the same.

Back NextHomepage



3/4 went to mental-health specialist. Patients who rated themselves worse at

outset made the most progress. AA very highly evaluated Therapy did as well as medications 40% got drugs;

MDs gave medications to 83% of patients MH pros; 20% drug treatments 50% who got drugs got no counseling 20% got no information about side effects 40% of anti-anxiety drugs given > 1 year

Consumer reports survey, 6.

Other effectiveness / descriptive findings:Other effectiveness / descriptive findings:

Back NextHomepage



Observation + Archival Controls

Dependent Variable(s):Combine survey or other measures with archival or ancillary data as Control variables.

More complex case studies

Basic selection / convenience biases, uncontrollable event.

Group Naturally occurring event

Example 1: San Francisco earthquake & coping

Sampling frame: - Randomly selected survey participants

Outcomes: - Standardized mental health scales- Self-reports of stress

Quasi-controls: - population norms on outcomes- ancillary measures, e.g., social support

Findings: - High rates of stress Rx, - Social support ‘buffers’ stress

Back NextHomepage



“One Shot” + archival control: examples

Event: - Three-mile Island nuclear accident

Sampling frame: - Randomly selected residents of geographic area around TMI

Outcomes: - Blood draws for immune markers- Self-reports of stress

Quasi-controls: - Demographically matched sample- Archival data on health & illness

Findings: - Long-term suppression of key immune markers (natural killer cells, T cells)

Example 2: Stress and immune functioning

Back NextHomepage



Event: - Loss of spouse

Sampling frame: - Hospital records, self-selected spouses

Outcomes: - Blood draws for immune markers- Standardized mental health scales- Occupational functioning

Quasi-controls: - Population norms on MH scales- Archival data: occupation & illness

Findings: - Long-term immune suppression- Social support ‘buffered’ stress- impact of bereavement > other stressors

One shot designs with archival controls, 3

Example 3: Psych. & Health effects of bereavement

Back NextHomepage



One shot designs; Summary

Virtue: Assess naturally occurring or uncontrollable socially or

politically important events

Provides “real world” look at processes that are typically studied in experiments: “Effectiveness” v. Efficacy data

Archival data can help interpret the findings / “control” some alternate interpretations.

Liability: lack of control group creates multiple threats to internal validity

No pre-measure makes interpretation (e.g., of change…) difficult.

“One Shot” designs: no control over independent variable(s), only partial control over measurement:

An experiment is not possible There cannot be a control group “Pre-” measures not possible or practical

Back NextHomepage





Measurement studies





Time series designs

Back NextHomepage



One group pre-test — post-test

Selected or convenience sample.

Group

Event or intervention May or may not be controllable by researcher, e.g., policy change.

Intervention or event Observe2

Outcome AssessmentTypically controllable, but may be archival.

Observe1

Baseline AssessmentMay or may not have control over measures (e.g., surveys v. archival measures).

Educational & social environments

Political or health policy change

Not feasible to have a control group

System-wide intervention / social change (school, public health campaign..)

Uses:

Back NextHomepage



Key design feature: no control group.

Maturation

Reactive measures

Statistical regression

Mortality / drop-out

History Historical / cultural events occur between baseline & follow-up.

Individual maturation or growth occurs between baseline & follow-up.

People respond to being measured or being a measured a second time.

Extreme scores at baseline “regress” to a more moderate level over time.

People leave the experiment non-randomly (i.e., for reasons that may affect the results…).

Group Intervention or event Observe2Observe1

Confound Observe2Observe1

Threats to internal validity (confounds):

Back NextHomepage



Examples: One group pre- post-, HIV testing

Event: - Receipt of HIV testing & counseling

Sampling frame: - Participants in testing centers

Study structure: - Baseline retrospective interview at testing session

- Follow-up interview 3 months later

Quasi-controls: - Population characteristics to predict between-group differences

Outcomes: - Self-reports of sexual risk

Example 1: Effects of HIV testing on sexual risk.

Back NextHomepage



Example: One group pre- post-, HIV, 2

Findings:

Effects of HIV testing on sexual risk, cont.

Threats to internal validity

- Self-selection into testing group

- Mortality: non-random drop-out(?)

- History: general shift in norms & behavior during study time may account for observed change

- Instrument change; people may answer more conservatively during a follow-up interview

- Significant shifts toward safety

- Few demographic predictors of risk or risk change

Back NextHomepage



Examples: One group pre- post- , Education

Intervention: - Standardized testing becomes integral to educational programs & school evaluation.

Sampling frame & - Longitudinal data across multipleStudy structure: years in target

school grades.

- No control group possible.

Quasi-controls: - Population characteristics to predict between-group differences

Outcomes: - Standardized test scores

Example 2: Educational reform & “No Child left behind” testing requirements.

Back NextHomepage



Example: One group pre- post-, education, 2

Findings: Modest, statistically significant increase in scores

Usual demographic predictors of change; more affluent, better schools..

Internal validity?: Reactive measures; teachers & students do better

when measured; (they also cheat; see Houston Miracle article)

Instrumentation: kids get better at taking standardized tests, teacher better at teaching them

History: General cultural shift Education more prominent in city

More affluent families sending kids to public schools

Education reform & test scores.

Back NextHomepage



One group pre- post- designs; Summary

Virtues: provide data on naturally occurring socially or politically

important events Pre-measure allows researcher to interpret change & examine

status of groups at baseline.

History maturation statistical regression reactive measures mortality / drop-out

One group pre- post- test design useful where: An experiment is not possible There cannot be a control group Researchers have control over measurement and the

independent variable

Liability: lack of control group creates multiple threats to internal validity:

Back NextHomepage



Groups are not equivalent at baseline, due to..

Self-selection Non-random assignment Use of existing groups Participants not blind

#1; Static Group Design

Non-equivalent two-group designs

Group1

Group2

Observe1

Observe1

Assessments may or may not be controlled

Survey or interviews Archival / existing data,

e.g., clinic records, grades

Intervention or event

Intervention or event may or may not be controlled by researcher;

Existing program Experimental intervention Naturally occurring event (..9/11..)

(No baseline)Contrast group

Back NextHomepage



Non-equivalent designs; pre- post-

Observation1 used to Assess equivalence of groups at baseline Test for threats to internal validity:

Reactive measuresHistory, mortality effectsRegression effects

Group

Group

Observe1

Observe1

#2 Two Group Pre- Post- Design

Non-equivalent groups Self-selection Non-random assignment Use of existing groups Participants not blind

Observe2

Intervention or event Observe2

Intervention & Assessments often controlled by researcher in these designs.

Similar to true experimental

design, except for non-equivalent

groups

Contrast group

Back NextHomepage



Examples: Non-equivalent groups, condoms

Study structure: - NY = intervention schools, Chicago are contrast schools.

- Baseline, sexual health programming, end of year Follow-up

Example Non-equivalent control group design: Effects of condom distribution on sexual safetyIntervention: - Condom education & distribution in

High School health classes

Outcomes: - Clinical measures: STDs

- Self-reports: sexual activity & safety

Sampling frame: - Schools in New York & Chicago

- Schools matched for SES, race, size

Back NextHomepage



Examples: Non-equivalent groups, condoms, 2

Findings: NY (intervention) students; lower STD rate, safer sex NY and Chicago students; similar levels of sexual

activity Thus; sexual health classes appeared to increase

safety without increasing sexual activity.

Internal validity?: Reactive measures; Study is not blind; NY students

know they are the intervention group

Non-equivalent groups: Possible differences between cities = unmeasured confounds

Condom distribution, cont.

Back NextHomepage



Non-equivalent designs

Groups 1 & 2:Observation1 used to Assess equivalence of groups at baseline Test threats to internal validity

Groups 3 & 4:Post-test only tests for reactive effects of assessment Compare 1+2 versus 3+4 Test interaction of treatment group x pre- post- versus post- only

Group 2

Group 1

Observe1

Observe1

Soloman 4-group design

Observe2

Intervention Observe2

Contrast group

Group 4

Group 3

Observe2

Intervention Observe2

Contrast group

Back NextHomepage



Non-equivalent 2 group designs:

Summary Most common quasi-experimental approach. Used where:

Some form of control or contrast group is possible Groups cannot be equivalent:

Participants cannot be blind re: group assignment Random assignment not possible Must use existing or self-selected groups.

Virtue: Study natural / “real world” interventionsContrast group lessens major threats to internal

validityLiability: non-equivalent groups = possible confound.

Back NextHomepage





Measurement studies





Time series designs

Back NextHomepage



Group Measure1 M2 M3 M4 M5 M6…


Interrupted time series design

Intervention may be experimental or observed

Policy shift, e.g., educational policy

Uncontrolled event; e.g., 9/11/01, Media event

Assessments may be experimental or archival Successive cross-sectional surveys

Traffic data, clinic or crime reports, test scores

Test effect of intervention or event on ongoing series of measurements.

Back NextHomepage





Time series designs

Multiple baseline

Demonstrate highly stable effect

long-term crime ratesdisease prevalenceeconomic performance…

Show steady rate of change

Hypothesis; tested by: Shift in stable rate after

intervention Increase / decrease in rate

of change after intervention

Threats to internal validity: sensitive to very local history Single group possibly prey to confound

Advantage for internal validity Eliminates carryover effects of repeated measurement tests maturation, history, reactive measurement, etc

Back NextHomepage



Example of interrupted time series: Shift in Baboon culture.

Core question: Do baboon troops develop and transmit a learned “culture”?

Baseline: Long-term observational data on aggressiveness in a specific baboon troop.

Intervention: Tuberculosis outbreak due to infected food. Dominant / aggressive males fed first

are selectively infected are naturally culled from troop

Naturally occurring event in >20yr. ongoing field study.

Back NextHomepage



Baboon culture: findings

Quasi-controls: Parallel data from other baboon troops.

Outcome measures: Standardized indices of aggression & dominance behavior

Core finding: With dominant males gone,

remaining males showed more cooperative behavior

Enhanced cooperation was transmitted across generation, showing learned “culture”.

Back NextHomepage



Example: Interrupted time series data

Data: Archival records of HIV tests reported to CDC, collected monthly

Data show stable baseline over multiple observations Timing of intervention precise relative to data collection

Intervention: Magic reports infection on national TV. Uncontrollable, “naturally occurring” event Tests hypothesis re: modeling effects in health behavior

Finding: Initial spike in testing rates, followed by leveling off at higher base rate.

Initial increase expected Hypothesis tested by longer-term shift in base rate, available

due to archival time-series data Effect found for both genders.

The “Magic Johnson effect” on HIV testing

Back NextHomepage



Example of time-series data: “Magic” / HIV effect.

Tesoriero, J.M., Sorin, M.D., Burrows, K.A., LaChance-McCullough, M.L. (1995). Harnessing the heightened public awareness of celebrity HIV disclosures: “Magic” and “Cookie” Johnson and HIV testing. AIDS Education and Prevention, 232-250.

Magic’sAnnouncement

Low & variable baserate of testing

Initial spike

New, higher base rate

Multiple (monthly) measures.

Time-series data showing shift in HIV testing after Magic’s announcement.

Back NextHomepage



Multiple time series study

Group 1 Measure1 M2 M3 M4 M5 M6…

Group 2 Measure1 M2 M3 M4 M5 M6…

Multiple time series data

Groups typically formed by blocking variable measured post-hoc;

Health claims in NYC v. other cities post- 9/11/01

Younger v. older voting patterns post- Iraq invasion

Heterosexual v. gay HIV testing rates post- Magic Johnson media event.

Hypothesis; tested by interaction of blocking variable by repeated measure:

Is shift in stable rate ( rate of change) greater in one group than another?


Back NextHomepage



Blocking variables

Testing blocking variables in the HIV testing time-series data.

Core questions:Both heterosexuals and Ethnic minorities had low HIV

testing rates May feel HIV is not relevant to them – it is a “white gay” problem. They may lack resources or venues for testing.

Will having a prominent African-American Heterosexual disclose HIV+ status may change those perceptions?

Hypotheses:Heterosexuals will respond more strongly to the Magic

Johnson media event than will gay/bisexual men. African-American and Latino men and women will

respond most strongly.

Back NextHomepage


Psychology 242, Dr. McKirnanWeek 12-13, quasi-experimental designs.

Blocking variables: sexual orientation, 1.

High base-line and high variability in testing rates among men with risky partners, and IDUs.

Testing blocking variables: Gay / IDU data.

Gay / bisexual men show less variable, but generally lower baserates.

Risky men & IDUs slightly increase, with substantial variability.

Gay & bisexual men show no change.

Back NextHomepage



Blocking variables: sexual orientation, 2.

Testing blocking variables: Heterosexuals.

In contrast to gay / bisexual men or IDUs, heterosexual show an initially low baserate.

Followed by a large spike after the announcement

And a much higher new baserate.

The hypothesis that heterosexuals would be more affected by the “Magic” announcement was supported by the interaction of Time x the blocking variable of sexual orientation.

Back NextHomepage



Blocking variables: ethnic differences

African-Americans and Hispanics show low baserates and a high spike post-announcement

Testing blocking variables: Ethnic differences.

Both groups go back toward their baselines shortly post-announcement.

Back NextHomepage



Blocking variables: ethnic differences, 2.

HIV testing among Whites was similar to African-Americans & Hispanics at baseline,

They showed stable, much higher testing rate after Magic’s HIV announcement.

Ethnic differences: White participants.

Back NextHomepage



Summary: Blocking variables in time series data

A series of measures before & after an event allows us to clearly identify patterns of behavior, and to test group differences (via blocking variables).

The hypothesis that ethnic groups would differ was supported by interaction of Time x the blocking variable of ethnicity (but in a direction that was not predicted: Whites showed more change).

Back NextHomepage



Time series designs: Summary

Time series is most common with archival data: existing, standard records collected for other purposes.

Used where: The hypothesis concerns changes in long-term trends Typically an experiment cannot be run

Simple practicality or cost, e.g., health care issues Ethics; crime rates, rates of domestic violence, etc. The target events are not controllable.

Virtue:

Study natural / “real world” processes or interventions

Blocking variables – comparing time trends across groups -- lessens major threats to internal validity

Liability: lack of control = possible confound.

Back NextHomepage



Quick quiz

Researchers often use _____ to help interpret “single shot” surveys

A = paradigm change

B = measurement studies

C = experimental controls

D = retrospective measures

Back NextHomepage



Quick quiz, 2

B = Maturation

E = Reactive measures

D = Statistical regression

A = Mortality / drop-out

C = History

Historical / cultural events occur between baseline & follow-up.

Individual maturation or growth occurs between baseline & follow-up.

People respond to being measured or being a measured a second time.

Extreme scores at baseline “regress” to a more moderate level over time.

People leave the experiment non-randomly (i.e., for reasons that may affect the results…).

Match:

Back NextHomepage



Quick quiz, 2

B = Maturation




C = History People respond to being measured or being a measured a second time.

Match:

Back NextHomepage



Quick quiz, 3

B = Maturation




C = History

Growth or natural change between baseline & follow-up.

Match:

Back NextHomepage



Quick quiz, 4

B = Maturation




C = HistoryPeople leave the experiment non-randomly (i.e., for reasons that may affect the results…).

Match:

Back NextHomepage





Quick quiz 5

This is called a:

A = Threat to internal validity

B = Manipulation check

C = Multiple baseline

D = ..lot of work.

Back NextHomepage



Quasi-experiments; Summary

2. Evaluate existing groups or program(s) Single shot survey or measure of an intervention

With or without control variables

Non-equivalent / pre-existing groups Static group or 2 group pre- post- design

Time series designs, often with archival data

1. Study naturally occurring events that could not be brought into a lab or a true experiment.

Measurement studies


Trade off internal for external validity

Back NextHomepage


THANK YOU !!

psychology 242, dr. mckirnanresearch ethics quasi-experiments studying naturally occurring events ...

Documents