10/2/2011

1

Krishna Bista

10/04/2011

WHAT IS INTERNAL VALIDITY?

“Internal validity is a crucial measure in quantitative studies, where it ensures that a researcher’s experiment design closely follows the principle of cause and effect” (Shuttleworth, 2009)

Could there be an alternative cause, or causes, that explain my observations and results?”

2

SmokingLow-income groups

� Internal validity: Are the methods correct and the

results accurate?◦ Do the research conditions warrant the conclusions?

◦ Without internal validity results are un-interpretable.

� External validity: Are the findings generalizable beyond that

particular study?◦ To what extent can the results be generalized?

◦ To what populations, settings, treatment variables, and measurement variables?

(Lunenburg & Irby, 2008)

3

10/2/2011

2

Do tests really measure what student learning?

Do college GPAs accurately predict on the job success?

Content validity: CSEQ should measure college experience, not

depression!

Face validity: African Americans take a history test, but all of the

questions were written by Asians.

Criterion validity: Do a person’s reported income predict his or her credit score?

Construct validity: E.g. scores on a “marital satisfaction” scale should be negatively related to spouse abuse.

Predictive validity: Does the Suicide Probability Scale accurately

predict which adolescents are likely to attempt suicide?

4

� Cause and Effect

� Single-Group Design (variables)

� Multiple-Group Design (variables)

5

Temporal precedence

Covariation of cause and effect

No alternative explanations

Cause Effectthen

Time

if X, then Yif not X, then not Y

Program OutcomeCauses?

Alternativecause Alternative

cause

Alternativecause

Alternativecause

Adopted from Brown (2010)6

10/2/2011

3

Administerprogram

Measureoutcomes

X O

Measurebaseline

O

Alternativeexplanations

Alternativeexplanations

Adopted from Brown (2010) 7

O =Observation/MeasurementX= Treatment Intervention

Administerprogram

Measureoutcomes

Measurebaseline

Alternativeexplanations

Alternativeexplanations

X OO

OODo notadministerprogram

Measureoutcomes

Measurebaseline

Adopted from Brown (2010) 8

� History

� Maturation

� Testing

� Instrumentation

� Statistical regression

� Selection

� Research mortality

� Interactions w/ selection

(Brown, 2010; Krathwohl & Smith, 2005; Rudestam, & Newton, 2001)

9

10/2/2011

4

� Specific events (9/11,Tsumami, bomb, flooded

schools, flu), during the life of the experiment

� The longer the interval between the pretest and

posttest, the more viable this threat.

� Example: A Longitudinal Study of the Adaptation of

International Students in the United States. (Toni Falbo,

University of Texas at Austin, 2005)

10

� Changes in physical, intellectual, or emotional

characteristics of the participants

� In longitudinal studies (pre-test/pos-ttest/multi-test), for

instance, individuals grow older, become more

sophisticated, more/less motivated, become

bored/anxious

� Example: Asian Students’ Voices: An Empirical Study of

Asian Students' Learning Experiences at a New Zealand

University (Campbell & Li, 2007).

Changes in language ability, communication skills, and cross-cultural adaptations

11

� Also called “pretest sensitization,” this refers to the

effects of taking a test upon performance on a second

testing.

� Testing becomes a more viable threat to internal

validity as the time between pretest and posttest is

shortened.

� Example: Pre/Post/multi-6th grader -10th grader

Does the test measure factual information that can be

easily recalled or sth else?

12

10/2/2011

5

� Changes in the way a test or other measuring

instrument is calibrated that could account for results of

a research study (e.g. Missouri Test).

� This threat typically arises from unreliability in the

measuring instrument.

� Observer drift—being bored

� Poor survey construction

13

� Occurs when individuals are selected for an

intervention or treatment on the basis of extreme scores

on a pretest.

� Extreme scores are more likely to reflect larger

(positive or negative) errors in measurement.

� Results move towards the mean

14

� This can occur when intact groups are compared. The

groups may have been different to begin with.

� E.g. Does watching American Idol increase singing in

the shower?

15

10/2/2011

6

� Occurs when differential selection is confounded with

maturational effects.

� Example: dating behavior--girls who attended a dance

school and who did not attend

Dance/Nondance Group × Pubertal Status interaction was found.

Breast Development × Dance/Nondance Group interaction was found.

(Gargiulo, Attie, Brooks-Gunn, &Warren, 1987).

16

� The differential loss of individuals from treatment

and/or comparison groups.

� This is often a problem when research participants are

volunteers.

� Volunteers may drop out if they find it is time

consuming.

� Example: A Longitudinal Study of the Adaptation of

International Students in the United States. (Toni Falbo,

University of Texas at Austin, 2005)

17

Whenever measuring affects DV score

� Behavior and self-report (faking)/Judgeaphobia

� May divert attention from experiment instructions to

others

� As an example, three different classrooms—students

with how, low, and average skills (CSEQ—ESL vs. Graduate

Students)

18

10/2/2011

7

� Hide your identity (don’t call yourself a

psychologist/researcher)

� Be informal, friendly, put them at ease

� Control with self-report measures (peer-evaluation,

anonymity, confidentiality/bogus pipeline)(Rudestam & Newton, 2001)

19

Campbell, J., & Li, M. (2007). Asian students’ voices: An empirical study of Asian students' learning experiences at a New Zealand university. Journal of Studies in International Education, 12 (4), 375-396. doi:10.1177/1028315307299422

Brown, T. T. (2010). Research Design. HRH Labor Markets Course: Analyze and Plan Human Resources for Health Global Health Workforce Economic Network, University of California, Berkeley.

Gargiulo, J., Attie, I., Brooks-Gunn, J., &Warren, M. (1987). Girls' dating behavior as a function of social context and maturation. Developmental Psychology, 23 (5), 730-737

Lunenburg, F. C., & Irby, B. J. (2008). Writing a successful thesis or dissertation: Tips and strategies for students in the social and behavioral sciences. Thousand Oaks, California: Sage Publications, Inc.

Rudestam, K. E., & Newton, R. R. (2001). Surviving your dissertation: A comprehensive guide to content and process (2nd Ed.). Thousand Oaks, California: Sage Publications, Inc.

Shuttleworth, M. (2009). Internal Validity. Retrieved from Experiment Resources: http://www.experiment-resources.com/internal-validity.html

20