agenda group hypotheses validity of inferences from research inferences and errors types of validity...

Agenda Agenda

Group Hypotheses

Validity of Inferences from Research

Inferences and Errors

Types of Validity

Threats to Validity

Validity Validity

Inferences and Errors

Types of Validity

Threats to Validity

Inferential hazards Inferential hazards

Empirical tests are inherently ambiguous

Results cannot confirm or prove theory

Possible rival explanations persist

Conclusions about causal effects are clouded by a number of design factors

Lack of control over extraneous influences

Processes of measurement

Inferential hazards Inferential hazards

Two fundamental mistakes possible:

We conclude our theory is TRUE when in

truth it is FALSE (Type One Error)

We conclude our theory is FALSE when in

truth it is TRUE (Type Two Error)

Theory Is Actually:

True False

Conclude TheoryIs True

Conclude TheoryIs False

Correct

Correct

Type I Error

Type II Error

Validity Validity

Has to do with the “degree of doubt”

surrounding our inferences

Doubts about whether we are measuring what

we think we are

Doubts about whether we observe a

relationship

Doubts about whether the observed

relationship is causal

Doubts about whether we can generalize from

the relationship

X

Exposure to violent TV

Y

Aggressive Behavior

XViewing eithera boxing filmor sports film

YAdministration of shocks in a “learning” study

External Validity

Construct(Measurement)Validity

Internal Validity

Generalization to Population in the “Real World”Statistical

Conclusion Validity

?

Construct validity Construct validity

All measures imply a theory

All measures contain some error

Random error (“noise”)

Systematic error (“bias”)

How valid is our interpretation of a

measure?

Statistical conclusion validity

Statistical conclusion validity

Observed relationships depend upon:

Random processes

Number of observations

The “noisiness” (unreliability) of our measures

How certain are we that we have observed a

relationship?

Randomness raiseschance of Type II Error

Hypothesis Is Actually:

True False

Accept Hypothesis

Reject Hypothesis

Correct

Correct

Type I Error

Type II Error

Systematic mistakesraise chances of either I or II, depending

Internal validity Internal validity

Concluding that a relationship is causal

requires:

Covariation

Temporal ordering of observed cause and effect

“Non-spuriousness” i.e., elimination of rival

explanations

How certain are we of a cause-effect

relationship?

External validity External validity

Sound generalizations depend upon:

Representativeness of the sample

The observational setting

Representativeness of the processes

observed

How certain are we that generalizations

are warranted?

Threats to validity Threats to validity

Threats are rival explanations

For our measures

For why we did (or didn’t) observe a

relationship

For what caused the relationship

For the meaning of the relationship in

everyday life

Ambiguous Causal DirectionAmbiguous Causal Direction

Lack of established temporal order may render reverse causation plausible Common in correlational studies

1Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity

SelectionSelection

Systematic differences may be present in groups selected for comparison Groups must be made comparable

By random assignment (best)

By matching (not as good)


HistoryHistory

Changes in the environment may occur between measurements Example: Studying a community intervention


(A) X (health campaign)

(B) No X ( no campaign)

Y (diet)

Y (diet)

Other events could account for changes in Y

MaturationMaturation

People change naturally over time The longer the time between

measurements, the greater the possibility of maturation effects


Regression artifactsRegression artifacts

Extreme scores tend to “regress” toward the mean on a second test A problem if tests are used to select groups

and then repeated


First Time Scores

400510560600

...

Second Time Scores

240280310350

...

690730780800

280

350380

500520

490

640700760790

240

AttritionAttrition

People may drop out of studies May produce non-random group

comparisons


TestingTesting

Repeated measures can affect each other


(A) Y

(B) Y

Y

Y

Y (diet) Y (diet)

X (campaign)

X

No X

TestingTesting Measures can also interact with the manipulation

to produce effects (AKA “interaction of testing and treatment”)


(A) Y

(C)

Y

Y

X

X

(B) Y Y No X

(D) Y No X

InstrumentationInstrumentation

Changes in measurement processes can confound interpretation Instruments should be identical over time and across

groups “Ceiling” and “floor” effects may cloud interpretation of

measurements


Interactions with selectionInteractions with selection

Different groups maturing at different rates

Different groups experience “local history”

Different groups experience unique “ceiling” or “floor” effects on measures


Reactivity to experimental settingReactivity to experimental setting

People may respond, not to the independent variable per se, but to the situation Experimenter demand

1Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity

Compensatory equalizationCompensatory equalization

Administrative equity can spoil comparisons Relevant when interventions provide

desirable public goods


Compensatory rivalryCompensatory rivalry

Recognition of treatment can cause “control” groups to compensate


Resentful demoralizationResentful demoralization

Recognition of treatment can cause “control” groups to become despondent, “act up,” etc.


Diffusion of treatmentsDiffusion of treatments

Contact between treatment and control groups can spoil comparisons Possible in experimental interventions May be present in some quasi-experimental

comparisons


Interaction of causal relationship with unitsInteraction of causal relationship with units

The effects might only apply to the groups manipulated or observed Volunteers College students

1Threats to External ValidityThreats to External ValidityThreats to External ValidityThreats to External Validity

Interaction of causal relationship with outcomesInteraction of causal relationship with outcomes

The effects might only apply to particular, observed facets of complex phenomena

Fuller picture of effects might lead to different conclusions


Interaction of causal relationship with settingInteraction of causal relationship with setting

Experimental setting may be complex or artificial

Effects may be limited to a particular time or place


For Tuesday For Tuesday

Manipulation, observation, and control of Variables

Experiments

Schutt, Ch. 7 on experimental design

Shadish, Cook & Campbell, Ch. 8 on problems

and solutions in conducting experiments

http://www.asc.upenn.edu/courses/comm522/

agenda group hypotheses validity of inferences from research inferences and errors types of validity...

Documents