ADVANCED RESEARCH METHOD

Experiments

SOCI5013: Spring 2004

Introduction

• Experiments involve

• Taking action

• Observing the consequences of that action

• Simply put:

• Experimentation means select a group of subjects, do something to them, and observe the effect of what was done

Topics Appropriate to Experiments

• Research projects with limited and well-defined concepts and propositions

• Hypothesis testing

• Causation/explanatory analysis

• Not ideal for descriptive purpose

Topics

• How to design an experiment to discover a way to reduce prejudice against African Americans?

• Researchers selected a group of persons, determine their levels of prejudice

• Give those subjects some treatment over a period of time

• Then measure their level of prejudice again

The Classical Experiment

• Three components in an experiment

• Independent and dependent variable

• Pre-testing and post-testing

• Experimental and control group

Independent and Dependent Variable

• In experiments, independent variable takes the form of an experimental stimulus

• The stimulus is a dichotomous variable, having two attributes, present or not present

• Independent and dependent variables must be operationally defined. That is: researchers have to define them in terms of how to practically measure them

Reduce Prejudice

• In the previous example that researchers devise an experiment to discover ways to reduce prejudice

• Dependent variable: prejudice

• Independent variable: the treatment

Pre-testing and Post-testing

• Subjects are first measured in terms of a dependent variable (pre-testing)

• Subjects are then exposed to stimulus/treatment (independent variable)

• Finally, subjects are measured in terms of a dependent variable again (post-testing)

• Comparing results from pre-testing and post-testing, researchers can determine the effects

Bias

• Never reveal research purpose to subjects.• However, most times, subjects can see through to find out

purposes• An experiment to figure out ways to reduce prejudice will

fail to achieve its goal if• Subjects understand the purposes, and• Subjects respond to the post-testing not in line with how

they feel, but in line with what researchers want

Experimental and Control Groups

• Experimental Group Control Group

Measure Dependent Variable

Measure Dependent Variable

Stimulus

Measure Dependent Variable

Measure Dependent VariableC

C

Hawthorne Experiment

• What affect workers’ job satisfaction and productivity?

• An experiment at Western Electric Works in The Chicago Suburb of Hawthorne

• Improve work conditions by better lighting increases productivity and satisfaction

• However, worker productivity continues to improve after the light was turned dim

Medical Experiment

• Experimental patients receive real drugs under study

• Control patients receive placebo, a pseudo drug that is irrelevant to the cure

• Researchers compare the two groups after a time interval to determine the effect

Bias II

• Subjects, when figuring out the purpose, can fake the results

• Experimenters, likewise, are also inclined to report results consistent with expectations

• In a double-blind experiment, neither the experimenters nor the subjects know which one is the control group, which one is the experimental group

Selecting Subjects

• Cardinal rule of subject selection

• Experimental and control groups be as similar as possible.

• Thus any changes in post-testing comparison between the two groups are solo consequences of the treatment received by the experimental groups

Probability Sampling

• Random selected subjects from a population can be split into two groups: experimental and control groups

• The logic is: if both groups are representative samples of the population from which they are drawn, both groups should be similar to each other

Randomization

• Generalizability is not a main concern for experimental studies

• Pre-testing resemblance between control group and experimental group is the emphasis

• To achieve great similarity between the two groups, one needs

• A significant number of subjects in each group

• Random selection of subjects into each group

Matching

• Quota Matrix

White Male

Black Male

White Female

Black Female

Under 30 8 12 10 16

30-50 18 30 14 28

Over 50 12 20 12 22

Matching 2

• A desirable result• On the average both groups have the same age, the

same gender composition, the same racial composition, and so forth

• Quota variables may change• A general rule is both groups should be

comparable in terms of those variables that are most likely to be related to the dependent variable under study

Matching or Randomization

• Use randomization when you have large sample, because

• It is hard to know beforehand what those matching variables are

• Statistical technique assumes randomization

• When you have only a few cases, matching is better than randomization

Matching and Randomization

• Milton Yinger (1977) used both matching and randomization in an experiment with a large number of respondents

• First, homogenous strata are created according to 15 variables

• Second, complete random selection is used to split subjects in each strata into two equal half

Variations in Experimental Design

• One-shot case study

• One-group pretest-posttest design

• Static group comparison

One-shot Case Study

• A single group of subjects is measured on a dependent variable following the administration of some experimental stimulus

• A group of subjects were shown movies on African American History.

• In the end, they were measured regarding their prejudice.• Low level of their prejudice is attributed to their exposure

to the movie• Problem with this approach?

One-group Pretest-Posttest Design

• Determine prejudice level of the subjects in the beginning

• Show them films on African American history• Determine prejudice level of the subjects in the

end• The relative lower prejudice level in the end is

attributed to exposure to the films• Problems?

Static group comparison

• Add one control group

• Show the movie (African American History and Culture) to subjects in experimental group

• Do not show the movie to control group

• After a period of experiment, compare level of racial prejudice between the two groups

• Problems

Internal Invalidity

• Internal invalidity: possibility that the conclusions drawn from experimental results may not accurately reflect what has gone on in the experiment itself.

• 12 sources of internal invalidity

Sources of Internal Invalidity

• History: During the course of the experiment, historical events may occur that will confound the experimental results

• Maturation: people are growing and changing, such changes can affect the results of the experiment. Long-term versus short-term experience.

• Testing: the process of testing and retesting may influences people’s behavior, therefore confounding the experimental results.

Source of Internal Invalidity II

• Instrumentation: Use of different measuring items can induce differences due to measurement but not the experiment, thus confounding the results.

• Statistical regression: eliminate outliers

• Selection bias: Ensure that the two groups are comparable at the beginning of an experiment

• Experimental mortality: Sample attrition that causes sample bias

Sources of Internal Invalidity III

• Causal time order: independent variables have to be preceding of the dependent variable.

• Diffusion or imitation of treatments: contaminating the control group

• Compensation: in some medicine experiments, hospital staff feels sorry for the control group and offer extra tender loving care

• Compensatory rivalry• Demoralization

Solution• Experimental Group Control Group

Measure Dependent Variable

Measure Dependent Variable

Stimulus

Measure Dependent Variable

Measure Dependent VariableC

C

External Invalidity

• External invalidity relates to generalizability of experimental findings to the “real” world.

• Even if the results of an experiment are accurate effects of what happened during the experiment, do those experiment really tell us anything about life?

An Example

• Assume we have compelling evidence that subjects in the experimental group had lower their prejudice after seeing African American History movies

• Does that make a difference if those movies are shown in TV or theater?

• In other words, experimentees may react to scientific experiment different they react to entertainment or other settings

• This is called interaction between testing situation and experimental stimulus

Solomon Four-group Design

• Group 1

• Group 2

• Group 3

• Group 4

Pretest

Posttest

Posttest

Pretest Posttest

Posttest

S

S

Solomon Design

• To prove that exposure to African American History movies reduces prejudice, the followings are needed

• In Group 1: posttest prejudice should be less than pretest prejudice

• Posttest prejudice level in group 1 should be lower than group 2

• The Group 3 posttest prejudice should be lower than pretest prejudice in group 2

• The group 3 posttest should show less prejudice than the group 4 posttest prejudice

Posttest Only Control Group Design

• Campbell and Standley: with proper randomization, only Group 3 and 4 are needed for a true experiment that controls for the problem of internal invalidity as well as for the interaction between testing and stimulus

• G1

• G2 Posttest

PosttestS

Looking-Glass Self

• Charles Cooley and George Mead: how we behave are largely a function of how others see and treat us.

• Harvard Test of Inflected Acquisition: Researchers conducted initial “test” and identify those “good” students to their teachers

• In the following year, an IQ test on students found that those “good” students performed better than their classmates.

Looking-Glass Self 2

• However the initial test is a hoax. Researchers identify both “good” and “bad” students to their teachers

• Psychologists: expectations communication model: expectations of a dominant individual affect the performance of subordinates

• Sociologists: expectations-states theory: roles of expectations in subsequent performance among peers

Role Standard Experiment• Randomly select subjects into two groups:

experimental group and control group• To the experimental group, researchers conduct

so-called “pattern recognition ability” test• Subjects in experimental group received the test,

obtained their test grades, were informed of their partner’s grade, and received interpretation of their grades.

• However, the test was a fraud, the grades for both the subjects and their partners are computerized fictions, and the interpretations were manipulated by researchers.

Manipulations

• Researchers manipulate to produce scores and interpretations as the followings

• The subjects are definitely better in their cognitive skills than their partners

• The subjects are possibly better in their cognitive skills than their partners

• The subjects are possibly worse in their cognitive skills than their partners

• The subjects are definitely worse in their cognitive skills than their partners

Effects

• The control group receives the exam but is not graded on their performance, nor do they know their partner’s performance

• The second stage of the experiment involves pair-up subjects in experimental group with their partners and re-administer the test.

• During this stage, the subjects in experimental group make choices; then they were told what their partners have chosen; then they were allowed to change their answers

Results

• Then researchers count the number of times subjects change their answers

Group Mean Number of Changes

Definitely better 5.05

Possibly better 6.23

Control group 7.95

Possibly worse 9.23

Definitely worse 9.28

Result by Gender

• Split the results by gender

Group Women Men

Definitely better 4.50 5.66

Possibly better 6.43 6.10

Control group 7.68 8.34

Possibly worse 9.36 9.09

Definitely worse 10.00 8.70

Natural Experiment 1

• The behavioral consequences of suffering a natural disaster

• Following a devastating hurricane, researchers interview subjects in a town badly damaged by hurricane

• Then interview subjects in a town slightly damaged by hurricane

• Compare the results in terms of how subjects in each town prepare for the next possibly hurricane strike.

Natural Experiment 2

• Right after WWII,• Researchers interviewed residents living in several

German cities • some of the cities receive heavy bombing, others

receive light bombing• Then researchers compared subjects morale and

conclude that bombing did not reduce morale of German residents, a commonly assumption held by allies.

Natural Experiment 3• Right before the 1979 television show of

ROOTS by Alex Haley, researchers send questionnaire to residents in Washington state about their attitudes toward African Americans

• Right after the show, researchers send questionnaires to the same set of respondents asking how many of the show they have seen

• Subsequently, questionnaires were sent out to those respondents asking their attitudes toward African Americans

Strengths and Weaknesses

• Strengths

• Rigor in causality testing

• Involve little time and money

• Easily replicable

• Weakness

• Interactions between experiment and result