Scientific Method and Experiemental Design

Deduction v. Induction

All men are mortal

Socrates is a man.

Therefore Socrates is mortal.

--versus--

Is Socrates a Man?

Induction

• Sun came up yesterday

• Sun came up day before yesterday

• Sun came up the day before that

• …Sun will come up tomorrow

David Hume

• Deduction concerns “relations among ideas”

• Induction: big problems – In the past, the future has always been like the

past…– Is this circular logic?

Assumptions of Science

• Materialism – nature consists of matter that is observable

• The universe is mechanistic – there are discoverable cause-and-effect relations

• Determinism– Everything has a material cause.

Is it wrong to be a dualist?

• Could the mind interface the body through the pineal gland? YES.

• Is that question appropriate to science? NO.

The problem with “Creation Science”:

• Popper: Scientific theories are falsifiable.

Characteristics of science:

• Theories cannot be proved • Theories can only be disproved

Why do science then?

• Intuition can lead us astray

• Example: Perception is a leap of faith; our brain uses many shortcuts– Availability heuristic– Fundamental attribution error– Confirmation bias

Example: Clinical Judgment

• Paul Meehl (1920-2003)– “Clerks” better than psychiatrists

at predicting prisoner recidivism

• Medical treatment recommendations– Back surgery in Santa Barbara

vs. Bronx

• Hypothesis: a limited statement regarding cause and effect in specific situations– Dog barked because mailman approached

• Model: Mathematical (or algorithmic) statement that makes predictions over a limited range of situations.– Bark Rate = H x 1/Dmailman

• Theory: Barking caused by the approach of a unfamiliar organism.

Research methods

• Correlational

• Quasi-experimentation

• Experimentation

Correlational Research

“CASA research shows a statistical relationship between use of tobacco, alcohol and marijuana and use of drugs such as cocaine, heroin and acid. Among teens with no other problem behaviors, those who used cigarettes, alcohol and marijuana at least once in the past month are almost 17 times likelier to use another drug like cocaine, heroin and acid. Those who drank and smoked cigarettes at least once in the past month are 30 times likelier to smoke marijuana than those who did not.”

-The National Center on Addiction and Substance Abuse at

Columbia University

Correlation Causation

Pot use

Cocaine use

Pot use

Cocaine use

?Cocaine use

Pot use

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12

# Joints/week

mg

Co

cain

e/w

k

Correlation Coefficients

low high

low

high

low

high

low high

r = -1

r = +1

Research methods

• Correlational

• Quasi-experimentation:– Older adults lower “fluid intelligence” than

younger adults

• Experimentation

Quasi-experiments

• Used when it is impossible to manipulate IV

• Or when unethical to manipulate IV

• Problems:– Do groups differ beyond IV?

Research methods

• Correlational

• Quasi-experimentation:– Older adults lower “fluid intelligence” than

younger adults

• Experimentation

Experiment

• Experimenter manipulates independent variable and observes effects on the dependent variable

• Example: – IV: Drug A v. No Drug A– DV: Pain sensitivity

Experiment Design

• Between Subjects– Each subject experiences one level of the IV

• Within Subjects– Each subject experiences all levels of the IV

Between-Subjects Design

Within-Subjects Design

Drug A Test PlaceboTest

Placebo TestDrug ATestAll subjects

1/2

1/2

You want to see whether a GRE prep course is effective. You randomly select 50 students from Columbia and give them a GRE. Then you give the students the prep course and assess their GRE scores again at the end of the course. Their scores improved significantly. Is there a problem with this conclusion?

Research Issues• Internal validity: is the IV really responsible for

the effects on the DV? Are there confounds?– Extraneous variables: Other variables, in addition to the

IV, affect the DV.

• External validity: Does our experiment generalize to real situations?

• Construct Validity: Do our measures tap the intended construct.

• Reliability: Can this finding be replicated?• Expectancy: Did expectations of experimenter or

participant affect results?

Internal Validity Questions:

Did the Treatment Cause a Change in Behavior?

Research Issues• Internal validity: is the IV really responsible for

the effects on the DV? Are there confounds?– Extraneous variables: Other variables, in addition to the

IV, affect the DV.

• External validity: Does our experiment generalize to real situations?

• Construct Validity: Do our measures tap the intended construct.

• Reliability: Can this finding be replicated?• Expectancy: Did expectations of experimenter or

participant affect results?

External Validity Questions: Can the Results be Generalized?

• Can the results be generalized to other participants?

• Can the results be generalized to other settings?

Research Issues• Internal validity: is the IV really responsible for

the effects on the DV? Are there confounds?– Extraneous variables: Other variables, in addition to the

IV, affect the DV.

• External validity: Does our experiment generalize to real situations?

• Construct Validity: Do our measures tap the intended construct.

• Reliability: Can this finding be replicated?• Expectancy: Did expectations of experimenter or

participant affect results?

Construct Validity Questions:

Can We Make the Leap from the Physical World to the Mental

World?3 Follow-up Questions*

3 Key Construct Validity Questions

1. What does the measure really measure?

2. What does the manipulation really manipulate?

3. Is the participant’s behavior genuine or an act?

Research Issues• Internal validity: is the IV really responsible for

the effects on the DV? Are there confounds?– Extraneous variables: Other variables, in addition to the

IV, affect the DV.

• External validity: Does our experiment generalize to real situations?

• Construct Validity: Do our measures tap the intended construct.

• Reliability: Can this finding be replicated?• Expectancy: Did expectations of experimenter or

participant affect results?

Reliability: The statistics you obtain in your experiment don’t always equal the population

statistics.

5 4 3 2 2 1 6 7 4 5 9 8 7 5 4 5 3 2 1 5 5 4 7 8 4 3 4 5 9 1 3 7 4 5 6 4 2 1 3 5 6 3 7 5 3 4 2 5 1 9 5 3 4 5 8 7 7 1 5

The World

3 2 7 4 1 5

Your Sample

Research Issues• Internal validity: is the IV really responsible for

the effects on the DV? Are there confounds?– Extraneous variables: Other variables, in addition to the

IV, affect the DV.

• External validity: Does our experiment generalize to real situations?

• Construct Validity: Do our measures tap the intended construct.

• Reliability: Can this finding be replicated?• Expectancy: Did expectations of experimenter or

participant affect results?

Clever Hans

How to address validity issues:• Internal validity:

– Matching– Random assignment– Counterbalancing– Control Groups– Single- or double blind design

• External Validity– Random selection– Judicious selection of dependent measures and

independent variables

Example Experiment #1

• Study at CPMC: Expectant mothers in US randomly assigned to Groups: “Pray” or “No-Pray”

• People somewhere in Asia prayed for members of the “Pray” group. Double-blind study.

• Lower miscarriage rate in the “Pray” group.

Example Experiment #2

• Participants randomly assigned to two groups: “Stretch+Lift” or “Lift Only”

• All participants embark on 6 mo training regimen. Members of “S+L” group accompanied by trainer to guide stretching. “LO” group no trainer, no stretching.

• Bigger strength gains in the “S+L” group.

Example Experiment #3

• By J.P. Broca: Measured weight of 292 male and 140 female brains (at autopsy).

• Found that male brains averaged 1325g and female brains 1144g.

• Conclusion: Confirmation of male intellectual superiority.

Bias in intellectual assessment

Actual questions from the Army Alpha Test (and early intelligence test):

Crisco is a

a) patent medicine

b) disinfectant

c) toothpaste

d) food product

Christy Mathewson is famous as a(n)

a) writer

b) artist

c) baseball player

d) comedian

What’s missing?

Example Experiment #4

• Intro Psych “volunteers” randomly assigned to 2 groups: nasty feedback, no feedback.

• Nasty group performs academic exercise, then is given very unfavorable feedback about work.

• All S’s debriefed.• 1 year later, enduring effects are assessed.• Conclusion: Psych experiments don’t have

permanent detrimental effects.

Experiment #5

• Intended to assess effects of Supportive Therapy or Interpersonal Therapy on depression.

• Patients randomly assigned to SP or IPT. Some therapists did SP, some did IPT (30 therapists total). 12 weekly therapy sessions at times that were convenient for patients.

• Bigger reduction in depressive symptoms in IPT than SP group.