Validity and Reliability

Validity and Reliability

Illustration of Types of Evidence of Validity (Figure 8.1)

Reliability and Validity (Figure 8.2)

Reliability of Measurement (Figure 8.3)

Methods of Checking Validity and Reliability (Table 8.2)

Validity (“Truthfulness”)

Method ProcedureContent-related evidence Expert judgmentCriterion-related evidence Relate to another measure of the same variableConstruct-related evidence Assess evidence on predictions made from theory

Reliability (“Consistency”)

Method Content Time Interval ProcedureTest-retest Identical Varies Give identical instrument twiceEquivalent forms Different None Give two forms of instrumentEquivalent forms/retest Different Varies Give two forms of instrument, with

time interval betweenInternal consistency Different None Divide instrument into halves and

score each or use KRObserver agreement Identical None Compare scores obtained by two or

more observers

Reliability Worksheet (Figure 8.5)

Educational Research

Chapter 13Experimental Research

Gay and Airasian

Experimental Research

13

Experimental Research

Key Characteristics of Experimental Designs Procedures are designed that

address potential threats to validity Internal External Construct Statistical Conclusion

Statistical comparisons of different groups are conducted

Selecting Participants and Assigning Them to Treatments Decide on the experimental unit of

analysis to be treated individual group or groups organization

Randomly assign individuals to groups control for extraneous characteristics that might influence the outcome

Topics Discussed in this Chapter Defining characteristics The experimental process Manipulation and control Threats to validity

Internal validity External validity

Group designs Single subject designs

Defining Characteristics Research designed to investigate cause and

effect relationships through the direct manipulation of an independent variable and control of extraneous variables Independent variable – the variable being

manipulated Dependent variable – the variable in which the

effect of the manipulation of the independent variable are observed

Researcher manipulation and control – choice of treatments, choice of a research design, use of specific procedures, etc.

Experimental Process Selection and definition of problem Selection of participants & instruments

Selection and assignment of participants Selection of research plan

Comparison of two approaches Comparison of new versus existing approaches Comparison of different amounts of single approach

Execution of research plan Sufficient exposure to treatment Operationally different treatments

Analysis of data Formulation of conclusions

Manipulation and Control Manipulation

The researcher’s decisions related to what will make up the independent variable

Active variables versus assigned variables Control

The researcher’s efforts to remove the influence of any extraneous variables that might have an effect on the dependent variable

The goal is to be assured the only differences between groups is that related to the independent variable

Internal Validity

9

Experimental Validity

Internal validity – the degree to which the results are attributable to the independent variable and not some other rival explanation

External/ecological validity – the extent to which the results of a study are generalizable

Relative importance of internal and external validity

Internal Validity

Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression Differential selection of participants Mortality Selection-maturation interaction, etc.

Mortality Threat to Internal Validity (Figure 9.1)

Location Might Make a Difference (Figure 9.2)

Instrument Decay (Figure 9.3)

A Data Collector Characteristics Threat (Figure 9.4)

A Testing Threat to Internal Validity (Figure 9.5)

A History Threat to Internal Validity (Figure 9.6)

Could Maturation be at Work Here? (Figure 9.7)

The Attitude of Subjects Can Make a Difference (Figure 9.8)

Regression Rears Its Head (Figure 9.9)

Illustration of Threats to Internal Validity (Figure 9.10)

Techniques for Controlling Threats to Internal Validity (Table 9.1) Technique

Obtain More Obtain More ChooseStandardize Information Information Appropriate

Threat Conditions on Subjects on Details Design

Subject characteristics X X

Mortality X X

Location X X

Instrumentation X X

Testing X

History X X

Maturation X X

Subject attitude X X X

Regression X X

Implementation X X X

Threats to External Validity Pre-test treatment interaction Multiple treatment interference Selection treatment interaction Specificity of variables Treatment diffusion Experimenter effects Reactive arrangements

Artificial environment Hawthorne effect John Henry effect Placebo effect Novelty effect

Controlling for Extraneous Variables

Randomization Selection Assignment

Matching Identifying pairs of subjects “matched” on

specific characteristics of interest Randomly assigning subjects from each pair

to different groups Difficulty with subjects for whom no match

exists

A Randomized Posttest-Only Control Group Design, Using Matched Subjects (Figure 13.7)

Matching Process Based on Gender

ExperimentalGroup

ControlGroup

JohnJimJamesJoshJacksonJaneJohannaJulieJeanJeb

Controlling for Extraneous Variables Comparing homogeneous groups

Restricting subjects to those with similar characteristics

Problems related to restriction of generalization

Build the variable into the design (e.g., factorial design)

Using subjects as their own controls Multiple treatments across time Problem with carry-over effect

Analysis of covariance (ANCOVA)

Group Designs

Two major classes of designs Single-variable designs – one independent

variable Factorial designs – two or more independent

variables Three types of designs

Pre-experimental designs Experimental designs Quasi-experimental designs

Pre-Experimental Designs

Types One-shot case study One-group pretest-posttest design Static group comparison

Threats to internal validity – see Figure 13.1

Example of a One-Shot Case Study Design (Figure 13.1)

Example of a Randomized Posttest-Only Control Group Design (Figure 13.4)

Example of a Randomized Solomon Four-Group Design (Figure 13.6)

Example of a One-Group Pretest-Posttest Design (Figure 13.2)

Example of a Static-Group Comparison Design (Figure 13.3)

Example of a Randomized Posttest-Only Control Group Design (Figure 13.4)

True Experimental Designs

Types Pretest-posttest control group design Posttest only control group design Solomon four-group comparison

Threats to internal validity – see Figure 13.2

Example of a Randomized Pretest-Posttest Control Group Design (Figure 13.5)

Quasi-Experimental Designs

Types Non-equivalent control group design Time series design Counterbalanced design

Threats to internal validity – see Figure 13.2

Results (Means) from a Study Using a Counterbalanced Design(Figure 13.8)

Possible Outcome Patterns in a Time-Series Design(Figure 13.9)

Using a Factorial Design to Study Effects of Method and Class Size on Achievement (Figure 13.10)

Illustration of Interaction and No Interaction in a 2 by 2 Factorial Design (Figure 13.11)

Example of a 4 by 2 Factorial Design (Figure 13.13)

Effectiveness of Experimental Designs in Controlling Threats to Internal Validity (Table 13.1)

Subject Instru- Data Collec-Charac- Morta- Loca- ment tor Charac- Data Col- Matur- Atti- Regres- Implemen-

Design teristics lity tion Decay teristics lector Bias Testing History ation tudinal sion tation

One-shot casestudy – – – (NA) – – (NA) – – – – –

One group pre-posttest – ? – – – – – – – – – –

Static groupcomparison – – – + – – + ? + – – –

Randomized post-test-only controlgroup ++ + – + – – ++ + ++ – ++ –

Randomized pre-post-test controlgroup ++ + – + – – + + ++ – ++ –

Solomon four-group ++ ++ – + – – ++ + ++ – ++ –

Randomizedposttest onlycontrol group with matchedsubjects ++ + – + – – ++ + ++ – ++ –

Matching-onlypre-posttestcontrol group + + – + – – + + + – + –

Counterbalanced ++ ++ – + – – – ++ ++ ++ ++ –

Time-series ++ – + _ – – – – + – ++ –

Factorial withrandomization ++ ++ – ++ – – + + ++ – ++ –

Factorial without randomization ? ? – ++ – – + + + – ? –

KEY: (++) = strong control, threat unlikely to occur; (+) = some control, threat may possibly occur; (–) = weak control, threat likely to occur; (?) = can’t determine; (NA) = threat does not apply

Single-Subject Research Designs that can be applied when the sample

size is one Study behavior change in an individual as the result

of some treatment Subject serves as his or her own control

Rationale Sophistication of specific designs allows for the

control of internal validity threats Research is focused on therapeutic impact in clinical

settings, not contribution to a research base Group comparison designs are sometimes opposed

or unethical Group comparison designs are not possible

Single-Subject Research Concerns

External validity Low generalizability due to the nature of the design The effect of the baseline condition on the subsequent

effects of the treatment Threats can be lessened through replication

Internal validity Possible to control for most threats Repeated and reliable measures

Baseline stability Number of data points

Single-variable rule Specification or the nature and conditions of the

treatment

Single-Subject Research Designs

A B A Withdrawal A B A B Multiple Baseline Alternating Treatments

Data analysis and interpretation Based on visual inspection and analysis of a

graphic presentation of the results Criterion of effectiveness is clinical

significance, not statistical significance Debate about the use of statistical

procedures

Single-Subject Research Replication

Establishes the generalizability of findings Three types

Direct – same researcher with the same or different participants in a specific setting

Systematic – follows direct replication but with different researchers, behaviors, or settings

Clinical – development of treatment packages composed of tow or more interventions which have been found to be effective individually