Chapter 6: Research Validity

Continued10/2/2012

Road Map

• Turn in Reflection Assignment #1• Announcement about lab• Quick Review• Finish Chapter 6

About Lab

• Go to class• Turn in assignments• Starting group project this week—you can’t

contribute fully if you aren’t there!• Remember: can be dropped after 4 absences• If you fail lab, you fail the course• Pay close attention to detail

Quick Review

• Statistical Conclusion validity• Construct Validity– Threats: participant reactivity

experimenter effects (will discuss next)

Construct Validity

Criterion-Related Validity

Predictive Concurrent

DiscriminantConvergent

Content-Related Internal Structure

Threats to Construct Validity

• Factors that impact how well our operationalizations actually represent constructs

• Pg 171, Table 6.2 – long list of threats

• We will focus on two major ones:– Participant reactivity to the experimental situation– Experimenter effects

Participant Reactivity to the Experimental Situation

• Participants’ motives and perceptions

• Demand characteristics

• Positive self-presentation

Instruction set #1

We want to see how well you are able to learn the following sets of letters. Letters will appear in groups of 3 to 7, and each letter will appear on the screen for 1 second. Following the presentation of the letters, …

Instruction Set #2

In the following task, you will be presented with groups of letters ranging from 3 to 7 letters. Each letter will appear on the screen for one second. Your task is to…

Experimenter effects

• Researcher actions and characteristics that influence the responses made by the research participant

• Expectancies– Clever Hans

• Attributes– Biosocial– Psychosocial– Situational factors

Clever Hans

I Math

Internal Validity

• The extent to which we can accurately infer that the independent and dependent variables are causally related

Observed Effect (DV)

Independent Variable

Causally Related

Independent Variable

Observed Effect (DV)

Cause must precede effect

Cause and effect are related (covary)

No other explanation is plausibleNo other explanation is plausible

Special Considerations

• Extraneous variables

• Confounding extraneous variables

• Control

Threats to Internal Validity• History • Maturation • Instrumentation • Testing • Regression Artifact • Attrition • Selection • Additive and Interactive Effects

History

• Any event occurring after the study begins that could produce the observed outcome

• Differential history: only one group experiences history event

• Note what history is NOT

Maturation

• Changes in biological and psychological conditions that occur with the passage of time – Factors within the individual

• Example: Head Start program and achievement over a school year

Instrumentation

• Changes in the assessment/measurement of the dependent variable

• Example: multiple observers and interviewers

Testing

• Changes in a person’s score on the second administration of a test are a result of previously having taken the test

• Example: pre-test and post-test on memory task

Regression Artifact

• a.k.a. regression toward the mean

• The tendency for extreme scores to become less extreme on a second assessment

Attrition

• Participant drop-out– Don’t show up for appointment– Decide to discontinue study

• Differential attrition is especially problematic

Selection

• The choice of participants for the various treatment groups based on different criteria – NOT random assignment

Additive & Interactive Effects

• The combined effect of several threats to internal validity

External Validity

• The extent to which the results of the experiment can be generalized across variations in:– people– Settings– Treatments– Outcomes– Time

What Limits External Validity?

• Lack of random selection/sampling

• Chance factors

• IV-DV relationship varies across levels of another variable

5 main categories of external validity

• Population Validity• Ecological Validity• Treatment Variation Validity• Outcome Validity• Time Validity

Population Validity

• The extent to which the results of a study can be generalized to the larger population.

• Target population: the population to which we want to generalize the results

• Accessible population: the population of research participants available to the investigator

Ecological Validity• The extent to which the results of a study can be

generalized across settings or environmental conditions

• Which research setting receives the most criticism here?– laboratory

Temporal Validity• The extent to which the results of an experiment can be

generalized across time

Treatment Variation Validity

• The generalizability of results across variation of the treatment

Outcome Validity

• The generalizability of results across different but related dependent variables.

Balancing Internal and External Validity

• Inverse relationship

• The conditions that increase internal validity (control) decrease external validity