Laboratory v. field experiments

The tradeoff between internal and external validity

Natural sciences

• In natural sciences, most experiments have been undertaken in carefully controlled laboratory conditions– Seeking laws of physics, etc.• Inanimate objects are unlikely to react differently in

controlled and uncontrolled conditions• The main goal is to avoid contaminating the experiment

via the introduction of third variables• Far more exacting and finer measurement is possible

under controlled conditions

Laboratory experiments in social science

• Closely controlled social science experiments do have the advantage of limiting the impact of third variables, but the unnatural situation presents problems for applying the findings outside the laboratory– “Artificiality” of laboratory setting

• Human behavior is sensitive to the environment within which it occurs– People act differently in a laboratory than in the

natural world• Several characteristics of laboratories are thought to

influence behavior• The very third variables controlled in the lab may be

the ones that determine behavior in the real world

• So, findings from laboratory experiments may only be valid for laboratory environments

An example

• New commercials are tested in controlled conditions– Eye tracking– Liking for commercials– Influence on purchase interest

• May try to provide less artificial conditions for study– Simulated living room

• Commercials that test high in lab experiments often do not work very well when used in real marketing campaigns

So, experiments move out of the lab:

• Researchers want to retain some of the advantages of the experiment:– Ability to manipulate/introduce the independent

variable and to control how much of it is presented– Time order—which comes first

• While sacrificing some of their ability to control third variables

• The goal is to improve our ability to generalize our findings to the real world

The field experiment

• One way to do so is to carry out a ‘field experiment’– The researcher still manipulates the independent

variable, but she does so within the natural world– For example, police try randomly stopping

motorists and testing for alcohol to see if alcohol-related crashes decline

Information Resources, Inc.

• BehaviorScan system

What problems do we encounter?

• Greatly reduced ability to prevent third variable contamination of results– The crusading doctor in MHHP– History effects

• Cycles

• What can you do?– Measure/monitor likely alternative explanatory variables– Question subjects about sources of influence– Multiple manipulations of the independent variable over time– Multiple measurement waves– Multiple dependent measures

• None will be perfect

• Expense– Tradeoff between extensive and intensive study– Budget constraints on number of sites, etc.

• Access/permission– Some research may present concerns to

authorities, citizens, etc.• Gain authorization/support prior to entering the field• Maintain good relationships with community leaders,

etc. throughout the intervention/research

Capturing ‘natural experiments’

• Sometimes unusual or unique events occur– ALAR scare– Service disruptions– Political campaigns– School shooting– Institution of Wi-Fi in an area

“Natural experiments”

• Because most such events are unplanned, the ability to prepare for them is limited– May keep a research group, materials and resources

ready for certain types of events• Must engage in ‘firehouse research’ gathering as

much data as possible in a short time– Inefficient, and may miss important data– However, real-world events, etc. may provide very

valuable data—may have both internal and external validity

• Use statistics, research analysis models that are appropriate to experimental studies