Experimental Research Designs

Chapter 11

The general plan for carrying out a study where the independent variable is changed

Determines the internal validity Should provide for limited differences

between control and experimental groups

Experimental design

Controlling threats to internal validity◦ Preexperimental ◦ Quasi-experimental◦ Experimental

Number of independent variables◦ Single variable – one manipulated IV◦ Factorial – two or more IVs & at lease one IV is

manipulated

Classification of experimental designs

One group pretest-posttest ◦ Give a pretest that measures the DV◦ Administer the treatment◦ Give a posttest that measures the DV◦ Compare the pretest score to the posttest score◦ Validity concerns

History Maturation Instrumentation Regression

Preexperimental Designs

Static group comparison◦ Uses 2 or more preexisting intact or static groups◦ One is exposed to the treatment◦ The posttest is administered to both groups and

the results compared◦ Validity concerns

Non randomization – can’t generalize Selection bias Maturation Mortality

Preexperimental Designs

Nonrandomized control group (pre-post) Counterbalanced design Time-series with one group Time-series with a control group

Quasi-Experimental designs

Select two or more intact groups Use similar groups Randomly assign groups as experimental and control Give pretest to both groups Administer the treatment Give posttest to both groups Compare the test results and analyze Validity concerns

◦ Selection bias can be controlled with the pretest and ANCOVA◦ Interaction of selection and

maturation, regression or instrumentation

Nonrandomized control group (pre-post)

Something in the selection process that causes one group to possess a higher or lower level of maturity◦ Asking for volunteers

Interaction of selection & maturation

May occur if each group represented a different population

ANCOVA with the pretest scores as the covariate is the best way to analyze

Interaction of selection & regression

Ceiling effect of tests could cause students with high scores on the pretest to show little improvement

students with lower scores on the pretest to show more improvement

ANCOVA with the pretest scores as the covariate is the best way to analyze

Interaction of selection & instrumentation

Uses intact groups Groups rotate positions Contains a series of replications Use when several treatments need to be

studied Rotation helps remove differences between

groups.

Counterbalanced design

Experimental Treatments

Replication X1 X2 X3

1 Grp 1 Grp 2 Grp 3

2 Grp 2 Grp 3 Grp 1

3 Grp 3 Grp 1 Grp 2

Col. Mean Col. Mean Col. Mean

Periodic measurement on one group Introduces a treatment at some time period Looking for changes in the pattern when

treatment is added

One group time series

T1 T2 T3 T4 T5 T60

2

4

6

Example 1Example 2Example 3

time

Dep

en

den

t

X

Validity concerns◦ History

Strengths-repeated measures help rule out these threats if they don’t cause fluxuations◦ Maturation◦ Testing◦ regression

One group time series

Contains a control and experimental group Helps control history effect Can have multiple control and experimental

groups

Control group time series

T1 T2 T3 T4 T5 T60123456

Control Experimental

time

Dep

en

den

t

Randomized Subjects, Posttest-Only Control Group

Randomized Matched Subjects, Posttest-Only Control Group

Randomized Subjects, Pretest-Posttest Control Group

Solomon Three-Group Solomon Four-Group Simple Factoral

True Experimental Designs

Randomly assign participants to groups(30 or more per group)

Give experimental group the treatment Measure both on the dependent variable All other variables are held constant Uses randomization and a control group to

control the threats to internal validity Randomization ensures that initial differences

between groups are due to chance Controls for history, maturation, regression,

pretesting

Randomized Subjects, Posttest-Only Control Group Design

Use for ◦ Studying changing attitudes◦ Where pretests are not appropriate or available◦ Can include more that 2 groups

Possible threats are subject effects and experimenter effects

Mortality could be a threat because there is no pretest to know if those that dropout are different from those that stay

Can’t be used to measure change

Randomized Subjects, Posttest-Only Control Group Design

Similar to previous design Uses a matching procedure to create

equivalent groups◦ Matching variables should correlate to DV◦ Matched pair must be randomly assigned one to

the E group and one to the C.◦ Could use a pretest in this model◦ This procedure is useful for small groups (<30)

Threats to validity same as previous design

Randomized Matched Subjects, Posttest-Only Control Group

Randomly assign participants to the experimental and control groups

Give a pretest on the DV to both groups Administer the treatment to the experimental

group Give a posttest on the DV to both groups Compare the pre and post test results using a t-test

or F-test ANCOVA is preferred statistic Internal validity threat– test sensitizing Main concern is with external validity involving

interaction between the pretest and treatment.

Randomized Subjects, Pretest-Posttest Control Group

Use 3 groups Random assignment Same as randomized pretest-posttest,

control group Has second control group

◦ Not pretested◦ Exposed to treatment

Compares all three posttest scores Both control groups should be similar unless

there is test sensitizing

Solomon Three-Group

Use 4 groups Random assignment Same as randomized pretest-posttest, control group Has third control group

◦ Not pretested◦ Not Exposed to treatment

Compares all four posttest scores Two groups take the pretest and two do not Treatment is given to one pretest group and one none

pretested group Both control groups should be similar unless there is test

sensitizing Drawback is difficult to conduct and time consuming Compare the posttests using ANOVA

Solomon Four-Group