Educational Research

Chapter 10Experimental Research

Gay, Mills, and Airasian10th Edition

Topics Discussed in this Chapter Defining characteristics of

experimental research Conducting experimental research Manipulation and control Threats to validity

Internal validity External validity

Group designs

Defining Characteristics Research designed to investigate cause and

effect relationships through the direct manipulation of an independent variable and control of extraneous variables

Independent variable – the variable being manipulated

Dependent variable – the variable in which the effect of the manipulation of the independent variable is observed

Researcher manipulation and control – choice of treatments, choice of a research design, use of specific procedures, etc.

Experimental Process

Six steps to conducting experimental research Selection and definition of the problem

Statement of a hypothesis indicating a causal relationship between variables

Selection of participants and instruments Random selection of a sample of subjects from a

larger population Random assignment of members of the sample to

each group Selection of valid and reliable instruments

Experimental Process Six steps to conducting experimental

research (cont.) Selection of a research plan

Three types of comparisons Comparison of two different approaches Comparison of new and existing approaches Comparison of different amounts of a single

approach Execution of the research plan

Two concerns Sufficient exposure to the treatment They need to be substantively different treatments

Experimental Process Six steps to conducting

experimental research (cont.) Analysis of data Formulation of conclusions

Manipulation and Control Manipulation

The researcher’s decisions related to what constitutes the independent variable

Active and assigned variables Active variables are those the researcher actively

manipulates Choice of an instructional strategy A particular counseling approach

Assigned variables are those that cannot be manipulated by the researcher but are of interest

Gender Race

Manipulation and Control

Control The researcher’s efforts to remove the influence of any extraneous variables that might have an effect on the dependent variable

The goal is to be assured the only differences between groups is that related to the independent variable

Participant variables – characteristics of the subjects Pre-existing achievement levels Differences in attitudes

Environmental variables – characteristics of the context Learning materials Differences in the time available for treatment between

groups

Experimental Validity

Internal validity – the degree to which the results are attributable to the independent variable and not some other rival explanation

External validity – the extent to which the results of a study can be generalized Population validity – generalizations related to

other groups of people Ecological validity – generalizations related to

other settings, times, contexts, etc.

Experimental Validity

Relative importance of internal and external validity Internal and external validity are related

reciprocally Controlling internal validity decreases external validity Controlling external validity decreases internal validity

First demonstrate an effect in a highly controlled environment (i.e., prioritize internal validity)

Second replicate the study in a more realistic, natural setting (i.e., prioritize external validity)

Threats to Internal Validity History:an event occurs not related to IV Maturation: Ss change over time Testing: Exposure to pretest might improve scores

on posttest Instrumentation: Reliability, Validity, and not using

the same test Statistical regression: Regression to the mean Differential selection of participants: Groups might

be different outside of IV Mortality: Ss drop out of the study Selection-maturation interaction, etc.: Groups

grow at different rates not due to the IV

Threats to External Validity Pre-test treatment interaction: Taking the pretest impacts

the treatment itself. Multiple treatment interference: More than one treatment/

experiment performed (old impacts new) Selection treatment interaction: Who is in your sample

impacts the results. Specificity of variables: Not specific enough in the following

areas to replicate the study or know if generalizable Participants Operational definition of the treatment Operational definition of the dependent variable Specific times Specific circumstances

Treatment diffusion: Two groups talk to one another and share treatment information so that they are not in effect one group.

Threats to External Validity

Experimenter effects: Something about the experimenter changes the outcome of the DV.

Reactive arrangements: Something about the Ss changes the outcome of the DV. Some examples of this are: reaction to the environment, reaction to the attention from the researcher, placebo effect, and novelty effect.

Controlling for Extraneous Variables

Extraneous variables must be controlled to be able to attribute the effect to the treatment Group equivalency must be assured

Four major means to achieve control Randomization

Selection – controls for representation Assignment – controls for group

equivalency

Controlling for Extraneous Variables

Matching Identifying pairs of subjects “matched” on specific

characteristics of interest Randomly assigning subjects from each pair to

different groups Difficulty with subjects for whom no match exists

Comparing homogeneous groups Restricting subjects to those with similar

characteristics Restricting subjects results in problems related to

generalization

Controlling for Extraneous Variables

Using subjects as their own controls Multiple treatments across time Problem with carry-over effects

Analysis of covariance (ANCOVA) Statistically adjusting the posttest scores

for the subjects in each group for pretest differences that existed at the beginning of the study

Creates statistically equivalent groups

Controlling for Extraneous Variables

Other ways to control extraneous variables Holding variables constant

Using only males rather than males and females

Selecting teachers with only similar levels of experience

Selecting only one grade level Stipulating the specific length of a

treatment

Group Designs

Two major classes of group designs Single-variable designs – one independent

variable Factorial designs – two or more independent

variables Three types of experimental designs

Pre-experimental designs Experimental designs Quasi-experimental designs

Pre-Experimental Designs Three types (X or X1=treatment, 0=test,

X2= control) One-shot case study

X O One-group pretest-posttest design

O X O Static group comparison

X1 O

X2 O

Threats to internal validity – see Table 10.1

True Experimental Designs

Three types (r=random assignment, x=treatment, 0=test) Pretest-posttest control group design

R O X OR O O

Posttest only control group design R X O

R O

True Experimental Designs

Three types (cont.) Solomon four-group comparison

R O X OR O OR X OR O

Threats to internal validity – see Figure 10.1

Quasi-Experimental Designs

Groups may be randomly assigned, not Ss Three types

Non-equivalent control group design O X O

O O Time series design

O O O O X O O O O Counterbalanced design

O X1 O X2 O X3 O

O X3 O X1 O X2 O

O X2 O X3 O X1 O

Threats to internal validity – see Figure 10.2

Factorial Designs Two independent variables and one

dependent variable The effect of teaching strategy and gender

on students’ achievement The effect of a particular counseling

technique and the clients’ ethnicity on the success of the treatment

The effect of a specific coaching approach and children in three age groups on the ability to perform certain physical tasks

Factorial Designs Interaction

The degree to which changes in the dependent variable are different depending on the levels of each of the independent variables

A particular instructional strategy is more effective for males than females

A particular counseling technique is more effective when the ethnicity of the counselor and client are similar

Factorial Designs Interaction

Visually explained by a graph of performance of all levels of both independent variables

Parallel lines indicate no interaction Non-parallel lines indicate an interaction

See Figure 10.5 in your text

Factorial Designs Interaction

Visual presentation of a significant interaction

LECTURE

Large Lecture RoomSmall Lecture Room

Te

st

Sc

ore

s

14

12

10

8

6

4

TEST

Small Testing Room

Large Testing Room

Factorial Designs Interaction

Visual presentation of a non-significant interaction

LECTURE

Large Lecture RoomSmall Lecture Room

Te

st

Sc

ore

s

18

16

14

12

10

8

TEST

Small Testing Room

Large Testing Room